{"id":18034,"date":"2018-06-24T04:10:27","date_gmt":"2018-06-24T04:10:27","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=18034"},"modified":"2018-06-24T04:10:27","modified_gmt":"2018-06-24T04:10:27","slug":"hbase-mapreduce-integration","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/","title":{"rendered":"HBase MapReduce Integration | MapReduce Over HBase"},"content":{"rendered":"

HBase integration with\u00a0Hadoop<\/strong>\u2019s MapReduce framework is one of the great features of HBase<\/strong>. So, to learn about it completely, here we are discussing HBase MapReduce Integration in detail. <\/span><\/p>\n

Moreover, we will see classes, input format, mapper, reducer. Also, we will learn MapReduce over HBase in detail, to understand HBase MapReduce well.<\/span><\/p>\n

So, let’s start HBase MapReduce Integration.<\/p>\n

What is MapReduce?<\/h2>\n

In order to solve the problem of processing in excess of terabytes of data in a scalable way, MapReduce<\/strong> process was designed. <\/span><\/p>\n

However, to build such a system that increases in performance linearly with the number of physical machines added, there should be a proper way. Basically, this is what the main purpose of MapReduce.<\/span><\/p>\n

Moreover, by splitting the data located on a distributed file system, it follows a divide-and-conquer approach. Hence, the servers which are available can access these chunks of data and also can process them as fast as they can. <\/span><\/p>\n

However, we will have to consolidate the data at the end with this approach. So, MapReduce has this built right into it, again.<\/span><\/p>\n

\"MapReduce<\/a>

MapReduce Integration<\/p><\/div>\n

4 Classes in MapReduce<\/h2>\n
\"HBase<\/a>

Classes in MapReduce Integration<\/p><\/div>\n

Here in the above MapReduce process figure, all the classes which are involved in the Hadoop implementation of MapReduce, is shown, let\u2019s learn them in detail:<\/span><\/p>\n

i. InputFormat<\/h3>\n
\"MapReduce<\/a>

InputFormat in HBase MapReduce Integration<\/p><\/div>\n

At very first, InputFormat splits the input data and further returns a RecordReader instance which defines the classes of the key and value objects. Also, it helps to iterate over each input record, with the help of next() method.<\/span><\/p>\n

ii.\u00a0Mapper<\/h3>\n

Now, by using the map() method, each record read using the RecordReader is processed, in this step.<\/span><\/p>\n

\"MapReduce<\/a>

HBase MapReduce Integration Mapper<\/p><\/div>\n

iii.\u00a0Reducer<\/h3>\n

This stage is as same as Mapper stage. Here we use to process the output of a Mapper class after shuffling and sorting of data.<\/span><\/p>\n

\"MapReduce<\/a>

Reducer in HBase MapReduce Integration<\/p><\/div>\n

iv.\u00a0OutputFormat<\/h3>\n

Finally, OutputFormat class hold the data in various locations. Here are some specific implementations which allow output to files, or in the case of the TableOutputFormat class to HBase tables. Moreover, to write the data into the specific HBase output table, it uses a TableRecord Writer.<\/p>\n

\"MapReduce<\/a>

HBase MapReduce OutputFormat Hierarchy<\/p><\/div>\n

Supporting Classes in MapReduce Integration<\/h2>\n

Now, in setting up MapReduce jobs<\/strong> over HBase, the MapReduce support comes with the TableMapReduceUtil class. There are some static methods which help to configure a job, hence we can run it with HBase as the source and\/or the target.<\/span><\/p>\n

MapReduce Over HBase<\/h2>\n

a. Preparation<\/h3>\n

In order to run a MapReduce job which needs classes from libraries, we\u2019ll need to make such libraries available before the execution of job only. Although, we have two choices, such as:<\/span><\/p>\n

    \n
  • Static preparation of all task nodes.<\/span><\/li>\n
  • Supplying everything needed for the job.<\/span><\/li>\n<\/ul>\n

    i. Static Provisioning<\/span><\/h4>\n

    Here, it is very useful to install the JAR file(s) of that library (which is used often) locally on the task tracker machines. Tracked machines are those machines which run the MapReduce tasks. It is possible by following steps:<\/span><\/p>\n

      \n
    • At very first, make the copy of the JAR files into a common location on all nodes.<\/span><\/li>\n<\/ul>\n
        \n
      • Further, with full location into the hadoop-env.sh configuration file, into the HADOOP_CLASSPATH variable, add the JAR files:<\/span><\/li>\n<\/ul>\n
        # Extra Java CLASSPATH elements. Optional.\n# export HADOOP_CLASSPATH=\"<extra_entries>:$HADOOP_CLASSPATH\"<\/pre>\n
          \n
        • Afterward,\u00a0to make the changes effective, restart all task trackers.<\/span><\/li>\n<\/ul>\n
          As its name, we can say this technique is quite static, although make sure every update here needs a restart of the task tracker daemons.<\/span><\/p>\n
          ii. Dynamic Provisioning<\/span><\/h4>\n
          Basically, we use dynamic provisioning approach, while we need to provide different libraries to each job we want to run, or also if we want to update the library versions along with your job classes.<\/span><\/p>\n
          b. Data Source and Sink<\/h3>\n
          An HBase table can be the source or target of a MapReduce job, or also we can use it as both input and output or we can say, for the input and output types, the third kind of MapReduce template uses a table. <\/span><\/p>\n
          In addition, it also includes setting the TableInputFormat and TableOutputFormat classes into the respective fields of the job configuration.<\/span><\/p>\n
          So, this was all about HBase MapReduce Integration. Hope you like our explanation.<\/p>\n
          Conclusion: HBase MapReduce Integration<\/h2>\n
          Hence, we have learned all about, HBase MapReduce Integration. We saw the meaning of MapReduce, its classes. Moreover, we saw supporting classes in HBase MapReduce Integration. However, if any doubt occurs regarding integration in HBase MapReduce, feel free to ask in the comment section.<\/p>\n","protected":false},"excerpt":{"rendered":"
          HBase integration with\u00a0Hadoop\u2019s MapReduce framework is one of the great features of HBase. So, to learn about it completely, here we are discussing HBase MapReduce Integration in detail. Moreover, we will see classes, input...<\/p>\n","protected":false},"author":7,"featured_media":18703,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23],"tags":[2538,4030,5382,8547,8558,15820,15821],"class_list":["post-18034","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hbase","tag-classes-in-mapreduce","tag-does-hbase-use-mapreduce","tag-hbase-mapreduce-integration","tag-mapreduce-integration","tag-mapreduce-over-hbase","tag-what-is-mapreduce","tag-what-is-mapreduce-integration"],"yoast_head":"\nHBase MapReduce Integration | MapReduce Over HBase - DataFlair<\/title>\n<meta name=\"description\" content=\"HBase MapReduce integration Tutorial,what is MapReduce integration,classes in MapReduce, supporting classes in MapReduce Integration, MapReduce over HBase\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"HBase MapReduce Integration | MapReduce Over HBase - DataFlair\" \/>\n<meta property=\"og:description\" content=\"HBase MapReduce integration Tutorial,what is MapReduce integration,classes in MapReduce, supporting classes in MapReduce Integration, MapReduce over HBase\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-24T04:10:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/HBase-MapReduce-Integration-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"HBase MapReduce Integration | MapReduce Over HBase - DataFlair","description":"HBase MapReduce integration Tutorial,what is MapReduce integration,classes in MapReduce, supporting classes in MapReduce Integration, MapReduce over HBase","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/","og_locale":"en_US","og_type":"article","og_title":"HBase MapReduce Integration | MapReduce Over HBase - DataFlair","og_description":"HBase MapReduce integration Tutorial,what is MapReduce integration,classes in MapReduce, supporting classes in MapReduce Integration, MapReduce over HBase","og_url":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-06-24T04:10:27+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/HBase-MapReduce-Integration-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"HBase MapReduce Integration | MapReduce Over HBase","datePublished":"2018-06-24T04:10:27+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/"},"wordCount":792,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/HBase-MapReduce-Integration-01-1.jpg","keywords":["classes in MapReduce","Does HBase use MapReduce","HBase - MapReduce Integration","MapReduce Integration","MapReduce Over HBase","what is MapREduce","what is MapReduce integration"],"articleSection":["HBase Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/","url":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/","name":"HBase MapReduce Integration | MapReduce Over HBase - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/HBase-MapReduce-Integration-01-1.jpg","datePublished":"2018-06-24T04:10:27+00:00","description":"HBase MapReduce integration Tutorial,what is MapReduce integration,classes in MapReduce, supporting classes in MapReduce Integration, MapReduce over HBase","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/HBase-MapReduce-Integration-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/HBase-MapReduce-Integration-01-1.jpg","width":1200,"height":628,"caption":"HBase MapReduce Integration | MapReduce Over HBase"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/hbase-mapreduce-integration\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"HBase Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/hbase\/"},{"@type":"ListItem","position":3,"name":"HBase MapReduce Integration | MapReduce Over HBase"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18034","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=18034"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18034\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/18703"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=18034"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=18034"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=18034"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}