- This topic has 3 replies, 1 voice, and was last updated 5 years, 7 months ago by
.
-
Posts
-
-
-
In MapReduce, jobs are broken into tasks and the tasks are run in parallel to make the overall job execution time smaller than it would otherwise be if the tasks ran sequentially. Now among the divided tasks, if one of the tasks take more time than desired, then the overall execution time of job increases.
Tasks may be slow for various reasons:
Including hardware degradation or software misconfiguration, but the causes may be hard to detect since the tasks may be completed successfully, could be after a longer time than expected. Apache Hadoop does not fix or diagnose slow-running tasks. Instead, it tries to detect when a task is running slower than expected and launches another, equivalent task as a backup (the backup task is called as speculative task). This process is called Speculative execution in MapReduce.Speculative execution in Hadoop does not imply that launching duplicate tasks at the same time so they can race. As this will result in wastage of resources in the cluster. Rather, a speculative task is launched only after a task runs for the significant amount of time and framework detects it running slow as compared to other tasks, running for the same job.
When a task successfully completes, then duplicate tasks that are running are killed since they are no longer needed.
If the speculative task after the original task, then kill the speculative task.
on the other hand, if the speculative task finishes first, then the original one is killed.
Speculative execution in Hadoop is just an optimization, it is not a feature to make jobs run more reliably.So if, I summarize:
The speed of MapReduce job is dominated by the slowest task. MapReduce first detects slow tasks. Then, run redundant (speculative) tasks. This will optimistically commit before the corresponding stragglers This process is known as speculative execution. Only one copy of a straggler is allowed to be speculated. Whichever copy (among the two copies) of a task commits first, it becomes the definitive copy, and the other copy is killed by the framework.Follow the link to learn more about Speculative execution in Hadoop
-
In MapReduce, a job is divided into tasks and executed in parallel across the cluster , which is faster compared to sequential execution.
In this parallel processing, there might be a chance for some/few of the tasks in some nodes to be slow compared to other tasks (in the other nodes) of the same job.
The reason can be anything: node busy, network congestion, etc, which limits the total execution time of the Job, and the system should wait for the slow running tasks to be completed.Here, Hadoop doesn’t try to fix the slow running tasks, instead clones the slow running tasks to the other nodes, where the rest of the tasks are completed, and gets executed there. This is termed as Speculative execution . As the name suggests, Hadoop tries to speculate the slow running tasks, and runs the same tasks in the other nodes parallely. Whichever task is completed first,that output is considered for proceeding further, and the slow running tasks are killed.
Hence, the MapReduce job is optimized with this phenomenon by optimizing the overall execution time of the Job.Speculative execution is enabled by default. To disable, edit mapred-site.xml conf. file by setting
mapreduce.map.speculative mapreduce.reduce.speculative to false.Follow the link to learn more about Speculative execution in Hadoop
-
MapReduce breaks jobs into Tasks that run in parallel manner rather than sequential, because of this the processing speed is fast for mapreduce. But there can be possibilities that a particular task might take more time to execute thus reducing the overall execution time of the job. Reasons that a Task might run slow can be Hardware problems or software Misconfigurations, so to overcome this we can execute a Duplicate Task to the original running task, this is called Speculative execution in Hadoop.
But if the original task is completed successfully then the Speculative task is Killed and Vice versa.
Speculative Execution enabled by default. To disable it we can edit mapred-site.xml configuration file:mapreduce.map.speculative mapreduce.reduce.speculative to false.
Follow the link to learn more about Speculative execution in Hadoop
-
- You must be logged in to reply to this topic.
