what is speculative execution in hadoop Map-Reduce

[DataFlair Team]

what is speculative execution in Map Reduce

[DataFlair Team]

In Hadoop, map-reduce breaks jobs into tasks and these tasks run parallel rather than sequentially which reduces overall execution time. But, if there are any slower tasks they slow down the overall execution time. Hadoop doesn’t diagnose these slow running tasks, instead, it tries to detect them and runs backup tasks for them.This process is called Speculative Execution.

Speculative execution is by default enabled in Hadoop. We can enable/disable the properties in the below file
Mapred-site.xml
<property>
<name>mapred.map.tasks.speculative.execution</name>
<value>false</value>
</property>
<property>
<name>mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
</property>
Duplicate job which was enabled as part of speculative process for a given will be killer if the original job is completed before it. Similarly Original job is killed if the duplicate job completed first

Follow the link to learn more about Speculative Execution

[Author]

Posts