This topic contains 1 reply, has 1 voice, and was last updated by  dfbdteam5 9 months, 4 weeks ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #5697

    dfbdteam5
    Moderator

    What is the difference between SparkSession and SparkContext in Apache Spark?
    Where should we use SparkSession / SparkContext?

    #5699

    dfbdteam5
    Moderator

    Spark Context:
    Prior to Spark 2.0.0 sparkContext was used as a channel to access all spark functionality.
    The spark driver program uses spark context to connect to the cluster through a resource manager (YARN orMesos..).
    sparkConf is required to create the spark context object, which stores configuration parameter like appName (to identify your spark driver), application, number of core and memory size of executor running on worker node.

    In order to use APIs of SQLHIVE, and Streaming, separate contexts need to be created.

    Example:
    creating sparkConf :

    val conf = new SparkConf().setAppName(“RetailDataAnalysis”).setMaster(“spark://master:7077”).set(“spark.executor.memory”, “2g”)
    
    creation of sparkContext:
    val sc = new SparkContext(conf)

    Spark Session:

    SPARK 2.0.0 onwards, SparkSession provides a single point of entry to interact with underlying Spark functionality and
    allows programming Spark with DataFrame and Dataset APIs. All the functionality available with sparkContext are also available in sparkSession.

    In order to use APIs of SQL, HIVE, and Streaming, no need to create separate contexts as sparkSession includes all the APIs.

    Once the SparkSession is instantiated, we can configure Spark’s run-time config properties.

    Example:

    Creating Spark session:
    val spark = SparkSession
    .builder
    .appName(“WorldBankIndex”)
    .getOrCreate()

    Configuring properties:
    spark.conf.set(“spark.sql.shuffle.partitions”, 6)
    spark.conf.set(“spark.executor.memory”, “2g”)

    Spark 2.0.0 onwards, it is better to use sparkSession as it provides access to all the spark Functionalities that sparkContext does. Also, it provides APIs to work on DataFrames and Datasets.

    For more details, please refer:
    SparkContext

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.