Speculative Execution in Apache Spark?

Viewing 2 reply threads
  • Author
    Posts
    • #6051
      DataFlair Team
      Moderator

      What is Speculative Execution in Spark?
      how to enable speculative Execution in Spark?
      Is there any drawback of speculative Execution?

    • #6052
      DataFlair Team
      Moderator

      > One more point is, Speculative execution will not stop the slow running task but it launch the new task in parallel.

      Tabular Form :

      Spark Property >> Default Value >> Description
      spark.speculation >> false >> enables ( true ) or disables ( false ) speculative execution of tasks.
      spark.speculation.interval >> 100ms >> The time interval to use before checking for speculative tasks.
      spark.speculation.multiplier >> 1.5 >> How many times slower a task is than the median to be for speculation.
      spark.speculation.quantile >> 0.75 >> The percentage of tasks that has not finished yet at which to start speculation.

    • #6055
      DataFlair Team
      Moderator

      The Speculative task in Apache Spark is task that runs slower than the rest of the task in the job.It is health check process that verifies the task is speculated, meaning the task that runs slower than the median of successfully completed task in the task sheet. Such tasks are submitted to another worker. It runs the new copy in parallel rather than shutting down the slow task.

      In the cluster deployment mode, the thread starts as TaskSchedulerImp1with spark.speculation enabled. It executes periodically every spark.speculation.interval after the initial spark.speculation.interval passes.

Viewing 2 reply threads
  • You must be logged in to reply to this topic.