- This topic has 2 replies, 1 voice, and was last updated 5 years, 7 months ago by DataFlair Team.
-
AuthorPosts
-
-
September 20, 2018 at 3:32 pm #5525DataFlair TeamSpectator
The map task copy is sent to all the blocks,that means same task copy is sent to all replicated blocks as well,so does it process all the replicated blocks also…this will consume more resources..how is it managed?
-
September 20, 2018 at 3:32 pm #5526DataFlair TeamSpectator
The Map task is sent to all the replicated blocks, but it does not process the Map tasks simultaneously across all the replicated blocks.
For example, if a block is replicated across three other blocks (named B1, B2, B3), then initially, B1 would start processing the Map task. Say supposing, if a node fails in a cluster which contains block B1, only then will the Map task start executing on the replicated block B2 and if the node containing block B2 fails, then the Map task will start running on block B3.
While the Map task is being run on any of the replicated blocks, the data team will work on bringing back the failed node alive so that the block B1 starts working and the replication factor of three is maintained across the clusters.To conclude, the HDFS data blocks are created across different nodes across the cluster and so the resource consumption need not be taken into consideration as there will be enough resources to facilitate the Map tasks on each node.
-
September 20, 2018 at 3:32 pm #5527DataFlair TeamSpectator
The map task is sent to all the blocks but it doesn’t process all blocks suppose if there are three duplicate(same) blocks it processes only one block.
While the Map taskis being run on any of the replicated blocks, the data team will work on bringing back the failed node alive so that the block B1 starts working and the replication factor of three is maintained across the clusters.
The HDFS data blocks are created across different nodes across the cluster and so the resource consumption need not be taken into consideration as there will be enough resources to facilitate the Map tasks on each node.
Learn More about : MapTask
-
-
AuthorPosts
- You must be logged in to reply to this topic.