This topic contains 1 reply, has 1 voice, and was last updated by  dfbdteam5 10 months ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #6013

    dfbdteam5
    Moderator

    Explain the Query Optimizer Framework of Spark.
    What is Catalyst query optimizer in Spark SQL?

    #6016

    dfbdteam5
    Moderator

    The most important component of Apache Spark is Spark SQL that deals with DataFrame API and SQL queries. Inside Spark SQL there lies an optimizer called Catalyst Query Optimizer. using this Spark creates an extensible query optimizer. This query optimizer Spark is based on Scala’s functional programming construct.

    Need of query optimizer:

    • To get solution to tackle various problem with Bigdata.
    • As a solution to extend the optimizer.

    We use catalyst general tree transformation frame work in four phases

    • Analysis
    • Logical Optimization.
    • Physical Planning.
    • Code generation.

    For more detailed study on Query optimization refer Catalyst Query optimizer in SparkSQL.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.