{"id":8182,"date":"2018-02-15T10:48:25","date_gmt":"2018-02-15T10:48:25","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=8182"},"modified":"2021-12-07T11:56:35","modified_gmt":"2021-12-07T06:26:35","slug":"sqoop-merge","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/","title":{"rendered":"Sqoop Merge – Tool to Combine Datasets in Sqoop"},"content":{"rendered":"

While it comes to combine two datasets we generally use merge tool in Sqoop<\/a><\/strong>. However, there are many more insights of Sqoop Merge available. So, in this article, we will learn the whole concept of Sqoop Merge. Also, we will see the syntax of Sqoop merge as well as arguments to understand the whole topic in depth. \u00a0\u00a0\u00a0<\/span>\u00a0\u00a0\u00a0<\/span><\/p>\n

So, let’s start Sqoop Merge tutorial.<\/p>\n

What is Sqoop Merge?<\/h3>\n

While it comes to combine two datasets we generally use merge tool in Sqoop. Especially, where entries in one dataset should overwrite entries of an older dataset. To understand it well, let\u2019s see an example of a sqoop merge.<\/span><\/p>\n

Here, an incremental import\u00a0<\/strong><\/a>run in last-modified mode will generate multiple datasets in HDFS<\/strong><\/a> where successively newer data appears in each dataset. Moreover, this tool will “flatten” two datasets into one, taking the newest available records for each primary key.<\/span><\/p>\n

Sqoop Merge Syntax & Arguments<\/h3>\n

$ sqoop merge (generic-args) (merge-args)
\n$ sqoop-merge (generic-args) (merge-args)
\nHowever, the job arguments can be entered in any order with respect to one another while the Hadoop<\/strong><\/a> generic arguments must precede any merge arguments.
\nTable. Merge options:<\/b><\/p>\n\n\n\n\n\n\n\n\n\n
Argument<\/b><\/td>\nDescription<\/b><\/td>\n<\/tr>\n
–class-name <class><\/span><\/td>\nSpecify the name of the record-specific class to use during the merge job.<\/span><\/td>\n<\/tr>\n
–jar-file <file><\/span><\/td>\nSpecify the name of the jar to load the record class from.<\/span><\/td>\n<\/tr>\n
–merge-key <col><\/span><\/td>\nSpecify the name of a column to use as the merge key.<\/span><\/td>\n<\/tr>\n
–new-data <path><\/span><\/td>\nSpecify the path of the newer dataset.<\/span><\/td>\n<\/tr>\n
–onto <path><\/span><\/td>\nSpecify the path of the older dataset.<\/span><\/td>\n<\/tr>\n
–target-dir <path><\/span><\/td>\nSpecify the target path for the output of the merge job.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Whenever this tool runs a \u00a0MapReduce<\/a><\/strong> job that takes two directories as input. Such as a newer dataset, and an older one. However, we can specify them with –new-data and –onto respectively. Moreover, MapReduce job’s output will be placed in the directory in HDFS<\/a><\/strong> specified by –target-dir.<\/p>\n

In addition,\u00a0here we assume that there is a unique primary key value in each record when we merge the datasets. Moreover, –merge-key specifies the column for the primary key. Also, make sure that\u00a0data loss may occur if multiple rows in the same dataset have the same primary key.<\/p>\n

Also, we must use the auto-generated class from a previous import to parse the dataset and extract the key column. Moreover, with –class-name and –jar-file we should specify the class name as well as jar file. Although, we can recreate the class using the codegen<\/a>\u00a0tool<\/a> <\/strong>if this is not available.<\/p>\n

To be more specific, this tool typically runs after an incremental import with the date-last-modified mode (sqoop import –incremental lastmodified \u2026).
\nNow let’s suppose that two incremental imports were performed, in an HDFS directory where some older data named older and newer data named newer. Hence, they could be merged as:<\/p>\n

$ sqoop merge –new-data newer –onto older –target-dir merged \\<\/p>\n

\u00a0\u00a0\u00a0–jar-file datatypes.jar –class-name Foo –merge-key id<\/p>\n

However, this would run a MapReduce job. Especially where we use the value in the id column of each row to join rows. Also, note that in preference to rows in the older dataset we use rows in the newer dataset.<\/p>\n

Basically, we can use it with both SequenceFile-, Avro- and text-based incremental imports. However, ensure that both the file types of the newer and older datasets\u00a0are same.<\/p>\n

So, this was all in Sqoop Merge tutorial. Hope you like our explanation.<\/p>\n

Conclusion<\/h3>\n

As a result, we have seen complete information on Sqoop Merge purpose, syntax, and Sqoop Merge Arguments. Still, if you feel any doubt regarding, feel free to ask through the comment section.<\/span>
\nSee also- Sqoop HCatalog<\/a><\/strong> & Sqoop Installation<\/strong><\/a>
\nFor reference<\/strong><\/a>

While it comes to combine two datasets we generally use merge tool in Sqoop. However, there are many more insights of Sqoop Merge available. So, in this article, we will learn the whole concept...<\/p>\n","protected":false},"author":9,"featured_media":8245,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[68],"tags":[4359,7138,8645,8646,10255,13661,13662,13663,13693,15985],"class_list":["post-8182","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sqoop","tag-example-of-sqoop-merge","tag-introduction-to-sqoop-merge","tag-merge-in-sqoop","tag-merging-in-sqoop","tag-purpose-of-sqoop-merge","tag-sqoop-merge-example","tag-sqoop-merge-purpose","tag-sqoop-merge-with-example","tag-sqoop-merge","tag-what-is-sqoop-merge"],"yoast_head":"\nSqoop Merge - Tool to Combine Datasets in Sqoop - DataFlair<\/title>\n<meta name=\"description\" content=\"What is Sqoop merge - Purpose of merge in sqoop, Sqoop merge Syntax and Sqoop Merge Arguments, Merging datasets in Sqoop with example\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/sqoop-merge\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Sqoop Merge - Tool to Combine Datasets in Sqoop - DataFlair\" \/>\n<meta property=\"og:description\" content=\"What is Sqoop merge - Purpose of merge in sqoop, Sqoop merge Syntax and Sqoop Merge Arguments, Merging datasets in Sqoop with example\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/sqoop-merge\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-15T10:48:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-12-07T06:26:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Introduction-to-sqoop-merge-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Sqoop Merge - Tool to Combine Datasets in Sqoop - DataFlair","description":"What is Sqoop merge - Purpose of merge in sqoop, Sqoop merge Syntax and Sqoop Merge Arguments, Merging datasets in Sqoop with example","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/","og_locale":"en_US","og_type":"article","og_title":"Sqoop Merge - Tool to Combine Datasets in Sqoop - DataFlair","og_description":"What is Sqoop merge - Purpose of merge in sqoop, Sqoop merge Syntax and Sqoop Merge Arguments, Merging datasets in Sqoop with example","og_url":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-02-15T10:48:25+00:00","article_modified_time":"2021-12-07T06:26:35+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Introduction-to-sqoop-merge-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/6e11a88eea98179e861a0933f8959f8f"},"headline":"Sqoop Merge – Tool to Combine Datasets in Sqoop","datePublished":"2018-02-15T10:48:25+00:00","dateModified":"2021-12-07T06:26:35+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/"},"wordCount":624,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Introduction-to-sqoop-merge-01-1.jpg","keywords":["example of sqoop merge","Introduction to sqoop-merge","merge in sqoop","merging in sqoop","Purpose of Sqoop Merge","Sqoop merge Example","Sqoop Merge purpose","Sqoop merge with example","sqoop-merge","what is Sqoop Merge"],"articleSection":["Sqoop Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/sqoop-merge\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/","url":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/","name":"Sqoop Merge - Tool to Combine Datasets in Sqoop - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Introduction-to-sqoop-merge-01-1.jpg","datePublished":"2018-02-15T10:48:25+00:00","dateModified":"2021-12-07T06:26:35+00:00","description":"What is Sqoop merge - Purpose of merge in sqoop, Sqoop merge Syntax and Sqoop Merge Arguments, Merging datasets in Sqoop with example","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/sqoop-merge\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Introduction-to-sqoop-merge-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Introduction-to-sqoop-merge-01-1.jpg","width":1200,"height":628,"caption":"What is Sqoop Merge"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/sqoop-merge\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Sqoop Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/sqoop\/"},{"@type":"ListItem","position":3,"name":"Sqoop Merge – Tool to Combine Datasets in Sqoop"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/6e11a88eea98179e861a0933f8959f8f","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/54363d8635c383dcd7e3be0ef00dbeef4217c45bf323d27bc2c92366b724df08?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/54363d8635c383dcd7e3be0ef00dbeef4217c45bf323d27bc2c92366b724df08?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/54363d8635c383dcd7e3be0ef00dbeef4217c45bf323d27bc2c92366b724df08?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"At DataFlair, our team of experts is committed to offering high-quality tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. We simplify learning to help you advance your career in the tech world.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam5\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8182","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=8182"}],"version-history":[{"count":5,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8182\/revisions"}],"predecessor-version":[{"id":105282,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8182\/revisions\/105282"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/8245"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=8182"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=8182"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=8182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}