- This topic has 4 replies, 1 voice, and was last updated 5 years, 7 months ago by
.
-
Posts
-
-
-
Yes number of Mappers can be changed in MapReduce job.There can be 100 or 1000 of mappers running parallelly on every slave and it directly depends upon slave configuration or on machine configuration on which the slave is running and these all slaves would be writing output on local disk.
So more the size or better the configuration of local machine on which slave is running more the number of mappers we can create.It is also to be noted that on each and every slave there can 100 or 1000 of Block and each block requires 1 mapper to process at any point in time.For Eg:-
If you have 1GB of the file that makes 8 blocks (of 128MB) so there will be only 8 mappers running on the cluster.
So based on file size also we can calculate the number of mappers required for processing. -
The number of Mappers for a MapReduce job is driven by number of input splits. And input splits are dependent upon the Block size.
For eg< If we have 500MB of data and 64MB is the block size in hdfs , then approximately number of mapper will be equal to 8 mappers.
and if we change the block size to 128MB, then the number of mapper will be approximately 4. -
The number of mappers is equal to the number of input splits over a lifetime of a map reduce job.
In the old API we have a job conf class which has a method setnumMaptaks(int) for setting the number of mappers
but in the new API, we use job class which doesn’t have such methods to define the number of maps.So the number of
mappers is equal to input splits.
JobTracker and Hadoop will take the responsibility of defining a number of mappers. -
In a Single word, no we cannot change the number of Mappers in MapReduce job but we can configure Reducers as per our requirement.
And the number of Mappers depends upon the number of InputSplit and this InputSplit depends on Size of your files and Block size. For example- If we have the block size of 128MB, then the number of mappers will be approximately 4.
-
- You must be logged in to reply to this topic.
