This topic contains 2 replies, has 1 voice, and was last updated by  dfbdteam3 1 year, 6 months ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #5558

    dfbdteam3
    Moderator

    What is the purpose of Mapper in Hadoop?
    What is Mapper in MapReduce?
    How Mapper works in Hadoop MapReduce?

    #5561

    dfbdteam3
    Moderator

    Mapper in Hadoop takes each record generated by the RecordReader as input. Then processes each record and generates key-value pairs. This key-value pair is completely different from the input pair. The mapper output is known as intermediate output which is stored on the local disk. Mapper does not store its output on HDFS, as it is temporary data and storing on HDFS will create multiple copies.

    Before storing mapper output on the local disk, partitioning of output takes place on the basis of the key and then sorting is done. This partitioning specifies that all the value for each key is grouped together. Mapper in hadoop only understands key-value pairs of data. So data should be converted into key-value pair before passing to the mapper. Data is converted into key-value pairs by InputSplit and RecordReader.

    1) InputSplit- InputFormat generates InputSplit which is the logical representation of data. MapReduce framework generates one map task for each Inputsplit.

    2) RecordReader- It communicates with InputSplit and converts data into key-value pairs.

    How many mappers?
    It depends on the total size of the input, i.e. the total number of blocks of the input files.

    Mapper= {(total data size)/ (input split size)}
    If data size= 1 Tb and input split size= 100 MB
    Mapper= (1000*1000)/100= 10,000
    Follow the link to learn more about Mapper in Hadoop

    #5563

    dfbdteam3
    Moderator

    1 ) What is the purpose of Mapper in Hadoop?

    In hadoop , mapper is used to convert the input split in key-value pairs .There will be the one mapper for each data block on HDFS.

    2 ) What is Mapper in MapReduce?

    Mapper is the user define program which manipulate the input split in (key, value) pairs as per the code design. Typically Mapper is the base class which need to extend by programmer to write their own logic as per requirement. While extending mapper, programmer need to mention input and output type under mapper class arguments.

    Example;

    Class MyMappper extends Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

    3 ) How Mapper works in Hadoop MapReduce?

    Mapper is the first code which is responsible to migrate/ manipulate the HDFS block stored data into key and value pair. Hadoop assign one map program to individually one blocks i.e. if my data is on 20 blocks then 20 map program will run parallel and the mapper output will getting store on local disk. After parsing the all data blocks content in key value pair by mapper, then hadoop serves this output to reducer as input. After reducer operation final output getting store in HDFS again.

    For more detail pease follow Mapper in Hadoop

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.