{"id":24822,"date":"2018-08-12T06:08:58","date_gmt":"2018-08-12T06:08:58","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=24822"},"modified":"2018-08-12T06:08:58","modified_gmt":"2018-08-12T06:08:58","slug":"hbase-compaction","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/","title":{"rendered":"HBase Compaction and Data Locality in Hadoop"},"content":{"rendered":"

In this Hadoop HBase tutorial of HBase Compaction\u00a0<\/span>and Data Locality with Hadoop, we will learn the whole concept of Minor and Major Compaction in HBase<\/strong>, a process by which HBase cleans itself in detail. <\/span><\/p>\n

Also, we will see Data Locality with Hadoop<\/strong> Compaction because data locality is a solution to data not being available to Mapper.\u00a0<\/span><\/p>\n

So, let’s start HBase Compaction and Data Locality in Hadoop.<\/p>\n

What is HBase Compaction?<\/span><\/h2>\n

As we know, for read performance, HBase is an optimized distributed data store. But this optimal read performance needs one file per column family. Although, during the heavy writes, it is not always possible to have one file per column family. <\/span><\/p>\n

Hence, to reduce the maximum number of disk seeks needed for read, HBase tries to combine all HFiles into a large single HFile<\/strong>. So, this process is what we call Compaction.<\/span><\/p>\n

In other words, Compaction in HBase is a process by which HBase cleans itself, whereas this process is of two types: Minor HBase Compaction as well as Major HBase Compaction.<\/span><\/p>\n

a. HBase Minor Compaction<\/h3>\n

The process of combining the configurable number of smaller HFiles into one large HFile is what we call Minor compaction. Though, it is quite important since, reading particular rows needs many disk reads and may reduce overall performance, without it.<\/span><\/p>\n

Here are the several processes which involve in HBase Minor Compaction, are:<\/span><\/p>\n

    \n
  1. By combining smaller Hfiles, it creates bigger Hfile.<\/span><\/li>\n
  2. Also, Hfile stores the deleted file along with it.<\/span><\/li>\n
  3. To store more data increases space in memory.<\/li>\n
  4. Uses merge sorting.<\/li>\n<\/ol>\n
    \"HBase<\/a>

    HBase Compaction<\/p><\/div>\n

    b. HBase Major compaction<\/h3>\n

    Whereas, a process of combining the StoreFiles of regions into a single StoreFile, is what we call HBase Major Compaction. Also, it deletes remove and expired versions. As a process, it merges all StoreFiles into single StoreFile and also runs every 24 hours.<\/span><\/p>\n

    However, the region will split into new regions after compaction, if the new larger StoreFile is greater than a certain size (defined by property).<\/span><\/p>\n

    Well, the HBase Major Compaction in HBase is the other way to go around:<\/span><\/p>\n

      \n
    1. Data present per column family in one region is accumulated to 1 Hfile.<\/span><\/li>\n
    2. All deleted files or expired cells are deleted permanently, during this process.<\/span><\/li>\n
    3. Increase read performance of newly created Hfile.<\/li>\n
    4. It accepts lots of I\/O.<\/span><\/li>\n
    5. Possibilities for traffic congestion.<\/span><\/li>\n
    6. The other name of major compaction process is Write amplification Process.<\/span><\/li>\n
    7. And it is must schedule this process at a minimum bandwidth of network I\/O.<\/li>\n<\/ol>\n
      \"HBase<\/a>

      HBase Major Compaction<\/p><\/div>\n

      HBase Compaction Tuning<\/span><\/h2>\n

      a. Short Description of HBase Compaction:<\/span><\/h3>\n

      Now, to enhance performance and stability of the HBase cluster, we can use some hidden HBase compaction configuration like below.<\/span><\/p>\n

      b. Disabling Automatic Major Compactions in HBase<\/span><\/h3>\n

      Generally, HBase users ask to possess a full management of major compaction events hence the method to do that is by setting HBase.hregion.majorcompaction<\/strong> to 0, disable periodic automatic major compactions in HBase.<\/span><\/p>\n

      However, it does not offer 100% management of major compactions, yet, by HBase automatically, minor compactions can be promoted to major ones, sometimes, although, we\u2019ve got another configuration choice, luckily, that will help during this case.<\/span><\/p>\n

      c. Maximum HBase Compaction Selection Size<\/span><\/h3>\n

      Control compaction process in HBase is another option:<\/span><\/p>\n

      hbase.hstore.compaction.max.size (by default value is set to LONG.MAX_VALUE)<\/strong><\/p>\n

      In HBase 1.2+ we have as well:<\/span><\/p>\n

      hbase.hstore.compaction.max.size.offpeak<\/strong><\/p>\n

      d. Off-peak Compactions in HBase<\/span><\/h3>\n

      Further, we can use off-peak configuration settings, if our deployment has off-peak hours.<\/span><\/p>\n

      Here are HBase Compaction Configuration<\/em> options must set, to enable off peak compaction:<\/span>
      \nhbase.offpeak.start.hour= 0..23<\/strong>
      \nhbase.offpeak.end.hour= 0..23<\/strong><\/p>\n

      Compaction file ratio for off peak 5.0 (by default) or for peak hours is 1.2.<\/span><\/p>\n

      Both can be changed:<\/span>
      \nhbase.hstore.compaction.ratio<\/strong>
      \nhbase.hstore.compaction.ratio.offpeak<\/strong><\/p>\n

      As much high the file ratio value will be, the more will be the aggressive (frequent) compaction. So, for the majority of deployments, default values are fine.<\/span><\/p>\n

      Data Locality in Hadoop<\/span><\/h2>\n

      As we know, in Hadoop, Datasets is stored in HDFS<\/strong>. Basically, it is divided into blocks as well as stored among the data nodes in a Hadoop cluster<\/strong>. Though, the individual Mappers will process the blocks (input splits), while a MapReduce job<\/strong> is executed against the dataset.<\/span><\/p>\n

      However, data has to copy over the network from the data node that has data to the data node that is executing the Mapper task, when data is not available for Mapper in the same node. So, it is what we call data Locality in Hadoop.<\/span><\/p>\n

      In Hadoop, there are 3 categories of Data Locality, such as:<\/span><\/p>\n

      \"HBase<\/a>

      Data Locality in Hadoop<\/p><\/div>\n

      1. Data Local Data Locality<\/span><\/h3>\n

      Data local data locality is when data is located on the same node as the mapper working on the data. In this case, the proximity of data is very near to computation. Basically, it is the highly preferable option.<\/span><\/p>\n

      2. Intra-Rack Data Locality<\/span><\/h3>\n

      However, because of resource constraints, it is always not possible to execute the Mapper on the same node. Hence at that time, the Mapper executes on another node within the same rack as the node that has data. So, this is what we call intra-rack data locality.<\/span><\/p>\n

      3. Inter-Rack Data Locality<\/span><\/h3>\n

      Well, there is a case when we are not able to achieve intra-rack locality as well as data locality because of resource constraints. <\/span><\/p>\n

      So, at that time we need to execute the mapper on nodes on different racks, and also then the data copy from the node that has data to the node executing mapper between racks. <\/span><\/p>\n

      So, this is what we call inter-rack data locality. Although, this option is less preferable.<\/span><\/p>\n

      So, this was all in HBase Compaction and Data Locality in Hadoop. Hope you like our explanation.<\/p>\n

      Conclusion: HBase Compaction\u00a0<\/span><\/h2>\n

      Hence, in this Hadoop HBase tutorial of HBase Compaction and Data Locality, we have seen the cleaning process of HBase that is HBase Compaction. A<\/span>lso, we have seen a solution to data not being available to Mapper<\/em>, Apache Hadoop Data Locality in detail. <\/span><\/p>\n

      Hope it helps! Please share your experience through comments on our HBase Compaction explanation.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

      In this Hadoop HBase tutorial of HBase Compaction\u00a0and Data Locality with Hadoop, we will learn the whole concept of Minor and Major Compaction in HBase, a process by which HBase cleans itself in detail....<\/p>\n","protected":false},"author":7,"featured_media":24912,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,23],"tags":[782,794,2702,3326,5238,5253,5254,5410,5411,5446,5456,15740],"class_list":["post-24822","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop","category-hbase","tag-apache-hadoop","tag-apache-hbase","tag-compaction-in-hbase","tag-data-locality-in-hadoop","tag-hadoop-data-locality","tag-hadoop-hbase","tag-hadoop-hbase-tutorial","tag-hbase-compaction","tag-hbase-compaction-tuning","tag-hbase-major-compaction","tag-hbase-minor-compaction","tag-what-is-hbase-compaction"],"yoast_head":"\nHBase Compaction and Data Locality in Hadoop - DataFlair<\/title>\n<meta name=\"description\" content=\"HBase Compaction & Data Locality in Hadoop, HBase Compaction tuning, HBase major compaction schedule, difference between HBase major and minor compaction\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/hbase-compaction\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"HBase Compaction and Data Locality in Hadoop - DataFlair\" \/>\n<meta property=\"og:description\" content=\"HBase Compaction & Data Locality in Hadoop, HBase Compaction tuning, HBase major compaction schedule, difference between HBase major and minor compaction\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/hbase-compaction\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-08-12T06:08:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/08\/HBase-Compaction-and-Data-Locality-in-Hadoop-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"HBase Compaction and Data Locality in Hadoop - DataFlair","description":"HBase Compaction & Data Locality in Hadoop, HBase Compaction tuning, HBase major compaction schedule, difference between HBase major and minor compaction","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/","og_locale":"en_US","og_type":"article","og_title":"HBase Compaction and Data Locality in Hadoop - DataFlair","og_description":"HBase Compaction & Data Locality in Hadoop, HBase Compaction tuning, HBase major compaction schedule, difference between HBase major and minor compaction","og_url":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-08-12T06:08:58+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/08\/HBase-Compaction-and-Data-Locality-in-Hadoop-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"HBase Compaction and Data Locality in Hadoop","datePublished":"2018-08-12T06:08:58+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/"},"wordCount":1036,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/08\/HBase-Compaction-and-Data-Locality-in-Hadoop-01.jpg","keywords":["apache hadoop","apache hbase","Compaction in Hbase","Data Locality in hadoop","Hadoop data locality","Hadoop HBase","Hadoop HBase tutorial","HBase Compaction","HBase Compaction tuning","HBase major compaction","HBase minor compaction","what is HBase compaction"],"articleSection":["Hadoop Tutorials","HBase Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/hbase-compaction\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/","url":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/","name":"HBase Compaction and Data Locality in Hadoop - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/08\/HBase-Compaction-and-Data-Locality-in-Hadoop-01.jpg","datePublished":"2018-08-12T06:08:58+00:00","description":"HBase Compaction & Data Locality in Hadoop, HBase Compaction tuning, HBase major compaction schedule, difference between HBase major and minor compaction","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/hbase-compaction\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/08\/HBase-Compaction-and-Data-Locality-in-Hadoop-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/08\/HBase-Compaction-and-Data-Locality-in-Hadoop-01.jpg","width":1200,"height":628,"caption":"HBase Compaction and Data Locality in Hadoop"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/hbase-compaction\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Hadoop Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/hadoop\/"},{"@type":"ListItem","position":3,"name":"HBase Compaction and Data Locality in Hadoop"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/24822","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=24822"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/24822\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/24912"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=24822"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=24822"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=24822"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}