{"id":1847,"date":"2017-04-04T14:18:33","date_gmt":"2017-04-04T14:18:33","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=1847"},"modified":"2018-11-14T17:13:35","modified_gmt":"2018-11-14T11:43:35","slug":"speculative-execution-in-hadoop-mapreduce","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/speculative-execution-in-hadoop-mapreduce\/","title":{"rendered":"Speculative Execution in Hadoop MapReduce"},"content":{"rendered":"

1. Objective<\/h2>\n

In this Big data<\/a><\/strong> Hadoop tutorial, we are going to learn Hadoop<\/strong> <\/a>speculative execution. Apache Hadoop does not fix or diagnose slow-running tasks. Instead, it tries to detect when a task is running slower than expected and launches another, an equivalent task as a backup (the backup task is called as speculative task). This process is called speculative execution in Hadoop.<\/p>\n

In this\u00a0tutorial we will discuss speculative execution – A key feature of Hadoop that improves job efficiency, what is the need of speculative execution in Hadoop, is Speculative execution always helpful or do we need to turn it off and how can we disable speculative execution in Hadoop.<\/p>\n

\"Speculative<\/a>

Speculative Execution in Hadoop MapReduce<\/p><\/div>\n

2. What is Speculative Execution in Hadoop?<\/h2>\n

Let us first understand what is Hadoop speculative execution?<\/p>\n

\"Speculative<\/a>

Speculative Execution in Spark<\/p><\/div>\n

In Hadoop, MapReduce<\/strong><\/a> breaks jobs into tasks and these tasks run parallel rather than sequential, thus reduces overall execution time. This model of execution is sensitive to slow tasks (even if they are few in numbers) as they slow down the overall execution of a job.
\nThere may be various reasons for the slowdown of tasks, including hardware degradation or software misconfiguration, but it may be difficult to detect causes since the tasks still complete successfully, although more time is taken than the expected time. Hadoop doesn\u2019t try to diagnose and fix slow running tasks, instead, it tries to detect them and runs backup tasks for them. This is called speculative execution<\/strong> in Hadoop. These backup tasks are called Speculative tasks in Hadoop.<\/p>\n

3. How Speculative Execution works in Hadoop?<\/h2>\n

Let us now see Hadoop speculative execution process.
\nFirstly all the tasks for the job are launched in Hadoop MapReduce. The speculative tasks are launched for those tasks that have been running for some time (at least one minute) and have not made any much progress, on average, as compared with other tasks from the job. The speculative task is killed if the original task completes before the speculative task, on the other hand, the original task is killed if the speculative task finishes before it.<\/p>\n

4. Is Speculative Execution Beneficial?<\/h2>\n

Hadoop MapReduce Speculative execution is beneficial in some cases because in a Hadoop cluster <\/strong><\/a>with 100s of nodes, problems like hardware failure or network congestion are common and running parallel or duplicate task would be better since we won\u2019t be waiting for the task in the problem to complete.
\nBut if two duplicate tasks are launched at about same time, it will be a wastage of cluster resources.<\/p>\n

5. How to Enable or Disable Speculative Execution?<\/h2>\n

Speculative execution is a MapReduce job optimization technique<\/strong><\/a> in Hadoop that is enabled by default. You can disable speculative execution for mappers<\/strong><\/a> and reducers<\/strong><\/a> in mapred-site.xml<\/em> as shown below:
\n[php]<property>
\n<name>mapred.map.tasks.speculative.execution<\/name>
\n<value>false<\/value>
\n<\/property>
\n<property>
\n<name>mapred.reduce.tasks.speculative.execution<\/name>
\n<value>false<\/value>
\n<\/property>[\/php]<\/p>\n

6. What is the need to turn off Speculative Execution?<\/h2>\n

The main work of speculative execution is to reduce the job execution time; however, the clustering efficiency is affected due to duplicate tasks. Since in speculative execution redundant tasks are being executed, thus this can reduce overall throughput. For this reason, some cluster administrators prefer to turn off the speculative execution in Hadoop.<\/p>\n

7. Conclusion<\/h2>\n

In conclusion, we can say that Speculative execution is the key feature of Hadoop<\/strong><\/a> that improves job efficiency. Hence, it reduces the job execution time. Hope you liked this blog, If you have any question regarding Speculative Execution in Hadoop, so please let us know by leaving a comment in a section given below. We will be glad to solve your queries.
\nSee Also-<\/strong><\/p>\n