How to configure hadoop to reuse JVM for mappers?

[DataFlair Team]

Every time a new Map/Reduce task is started, a new JVM is launched. As we know JVM launch is a costly process, how to configure hadoop to re-use the jvm of old mapper/reducer.

[DataFlair Team]

To configure Hadoop to reuse JVM for mappers, we just need to add entry in the configuration file: $HADOOP_HOME/conf/mapred-site.xml

<property>
<name>mapred.job.reuse.jvm.num.tasks</name>
<value>-1</value>
</property>

We need to specify a number value how many times the JVM is to be reused (default is 1), or one can set to -1 for no limit on the reuse of JVM.

[Author]

Posts