Free Online Certification Courses – Learn Today. Lead Tomorrow. › Forums › Apache Spark › Speculative Execution in Apache Spark?
- This topic has 2 replies, 1 voice, and was last updated 5 years, 6 months ago by DataFlair Team.
-
AuthorPosts
-
-
September 20, 2018 at 5:01 pm #6051DataFlair TeamSpectator
What is Speculative Execution in Spark?
how to enable speculative Execution in Spark?
Is there any drawback of speculative Execution? -
September 20, 2018 at 5:01 pm #6052DataFlair TeamSpectator
> One more point is, Speculative execution will not stop the slow running task but it launch the new task in parallel.
Tabular Form :
Spark Property >> Default Value >> Description
spark.speculation >> false >> enables ( true ) or disables ( false ) speculative execution of tasks.
spark.speculation.interval >> 100ms >> The time interval to use before checking for speculative tasks.
spark.speculation.multiplier >> 1.5 >> How many times slower a task is than the median to be for speculation.
spark.speculation.quantile >> 0.75 >> The percentage of tasks that has not finished yet at which to start speculation. -
September 20, 2018 at 5:02 pm #6055DataFlair TeamSpectator
The Speculative task in Apache Spark is task that runs slower than the rest of the task in the job.It is health check process that verifies the task is speculated, meaning the task that runs slower than the median of successfully completed task in the task sheet. Such tasks are submitted to another worker. It runs the new copy in parallel rather than shutting down the slow task.
In the cluster deployment mode, the thread starts as TaskSchedulerImp1with spark.speculation enabled. It executes periodically every spark.speculation.interval after the initial spark.speculation.interval passes.
-
-
AuthorPosts
- You must be logged in to reply to this topic.