- This topic has 2 replies, 1 voice, and was last updated 5 years, 7 months ago by
.
-
Posts
-
-
-
> One more point is, Speculative execution will not stop the slow running task but it launch the new task in parallel.
Tabular Form :
Spark Property >> Default Value >> Description
spark.speculation >> false >> enables ( true ) or disables ( false ) speculative execution of tasks.
spark.speculation.interval >> 100ms >> The time interval to use before checking for speculative tasks.
spark.speculation.multiplier >> 1.5 >> How many times slower a task is than the median to be for speculation.
spark.speculation.quantile >> 0.75 >> The percentage of tasks that has not finished yet at which to start speculation. -
The Speculative task in Apache Spark is task that runs slower than the rest of the task in the job.It is health check process that verifies the task is speculated, meaning the task that runs slower than the median of successfully completed task in the task sheet. Such tasks are submitted to another worker. It runs the new copy in parallel rather than shutting down the slow task.
In the cluster deployment mode, the thread starts as TaskSchedulerImp1with spark.speculation enabled. It executes periodically every spark.speculation.interval after the initial spark.speculation.interval passes.
-
- You must be logged in to reply to this topic.
