{"id":7369,"date":"2018-02-06T13:42:34","date_gmt":"2018-02-06T13:42:34","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=7369"},"modified":"2021-12-07T13:01:10","modified_gmt":"2021-12-07T07:31:10","slug":"sqoop-architecture-and-working","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/","title":{"rendered":"Apache Sqoop Architecture – How Sqoop works Internally"},"content":{"rendered":"

As we discussed the complete introduction to Sqoop in our previous article \u201c<\/span>Apache Sqoop \u2013 Hadoop Ecosystem Component<\/b><\/a>\u201d. In this article “Sqoop Architecture and Working”, we will learn about Sqoop Architecture. Also, we will learn to work with Sqoop to understand well. But before Sqoop architecture let\u2019s have a look at Sqoop introduction to brush up your Knowledge.<\/span><\/p>\n

What is Sqoop?<\/h3>\n

Basically, using RDBMS applications generally interacts with the relational database. Therefore it makes relational databases one of the most important sources that generate <\/span>Big Data<\/b><\/a>. However, in the relational structures, such data is stored in RDB Servers. Although, by offering feasible interaction between the relational database server and <\/span>HDFS<\/b><\/a>, Sqoop plays the vital role in Hadoop ecosystem.<\/span><\/p>\n

To be more specific, it is a tool that aims to transfer data between HDFS (Hadoop storage) and relational database servers. Such as MySQL, Oracle RDB, SQLite, Teradata, Netezza, Postgres and many more. In addition, imports data from relational databases to HDFS. Also,\u00a0exports data from HDFS to relational databases. <\/span><\/p>\n

Moreover, Sqoop can transfer bulk data efficiently between <\/span>Hadoop<\/b><\/a> and external data stores. Like as enterprise data warehouses, relational databases, etc.<\/span><\/p>\n

However, it is very interesting to know that this is how Sqoop got its name. <\/span>
\n \u201c<\/span>SQ<\/b>L to Had<\/span>oop<\/b> & Had<\/span>oop<\/b> to <\/span>SQ<\/b>L\u201d.<\/span>
\nMoreover, to import data from external datastores into <\/span>Hadoop ecosystem<\/strong><\/a> tools we use Sqoop. Such as <\/span>Hive<\/b><\/a> & HBase.<\/span>
\nSince we know what is Apache Sqoop now. Thus, let\u2019s understand Sqoop Architecture and Working now.<\/span><\/p>\n

Sqoop Architecture and Working<\/h3>\n

By using the below diagram, Let\u2019s understand Apache Sqoop 2 Architecture and how Sqoop works internally:<\/span><\/p>\n

\"Sqoop<\/a>

Sqoop Architecture<\/p><\/div>\n

Basically, a tool which imports individual tables from RDBMS to HDFS is what we call Sqoop import tool<\/strong><\/a>. However, in HDFS we treat each row in a table as a record.<\/span><\/p>\n

Moreover,\u00a0our main task gets divided into subtasks, while we submit Sqoop command. However, map task\u00a0individually handles it\u00a0internally. On defining map task, it is the subtask that imports part of data to the Hadoop Ecosystem. Likewise, we can say\u00a0all map tasks import the whole data collectively.<\/span><\/p>\n

\"Sqoop<\/a>

How sqoop Work internally<\/p><\/div>\n

However, Export also works in the same way.<\/span><\/p>\n

A tool which exports a set of files from HDFS back to an RDBMS is a Sqoop Export tool<\/strong><\/a>. Moreover, there are files\u00a0which behave as input to Sqoop which also contain records.\u00a0Those files what we call as rows in the table.<\/span>
\nMoreover, the job is mapped into map tasks,\u00a0while we submit our job, that brings the chunk of data from HDFS<\/strong><\/a>. Then we export these chunks to a structured data destination. <\/span><\/p>\n

Likewise, we receive the whole data at the destination by combining all these exported chunks of data.\u00a0However, in most of the cases, it is an RDBMS (MYSQL\/Oracle\/SQL Server).<\/span><\/p>\n

In addition, in case of aggregations, we require reducing phase. However, Sqoop does not perform any aggregations it just imports and exports the data. Also, on the basis of the number defined by the user, map job launch multiple mappers. <\/span><\/p>\n

In addition, each mapper task will be assigned with a part of data to be imported for Sqoop import. Also, to get high-performance Sqoop distributes the input data among the mappers equally.\u00a0Afterwards<\/span>, by\u00a0using JDBC each mapper creates the connection with the database. Also fetches the part of data assigned by Sqoop. Moreover, it writes it into HDFS or Hive<\/strong><\/a> or HBase<\/strong><\/a> on the basis of arguments provided in the CLI.<\/span><\/p>\n

Conclusion<\/h3>\n

As a result, we have seen the complete\u00a0Sqoop Architecture and its working. Still, if you feel to ask any query, feel free to ask in the comment section.
\nSee also-<\/strong>
\nSqoop List Databases<\/strong><\/a> &<\/strong>\u00a0Sqoop List Tables<\/strong><\/a>
\nFor reference<\/strong><\/a>

As we discussed the complete introduction to Sqoop in our previous article \u201cApache Sqoop \u2013 Hadoop Ecosystem Component\u201d. In this article “Sqoop Architecture and Working”, we will learn about Sqoop Architecture. Also, we will...<\/p>\n","protected":false},"author":9,"featured_media":35226,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[68],"tags":[1091,16657,13599,13600,13684,16267],"class_list":["post-7369","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sqoop","tag-architecture-of-sqoop","tag-how-sqoop-works","tag-sqoop-architecture","tag-sqoop-architecture-and-working","tag-sqoop-working","tag-working-of-sqoop"],"yoast_head":"\nApache Sqoop Architecture - How Sqoop works Internally - DataFlair<\/title>\n<meta name=\"description\" content=\"Sqoop Architecture-Learn Sqoop introduction,complete architecture of sqoop & how sqoop works internally.See Sqoop architecture Diagram & Sqoop import-export\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Sqoop Architecture - How Sqoop works Internally - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Sqoop Architecture-Learn Sqoop introduction,complete architecture of sqoop & how sqoop works internally.See Sqoop architecture Diagram & Sqoop import-export\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-06T13:42:34+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-12-07T07:31:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Sqoop-Architecture-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1202\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Sqoop Architecture - How Sqoop works Internally - DataFlair","description":"Sqoop Architecture-Learn Sqoop introduction,complete architecture of sqoop & how sqoop works internally.See Sqoop architecture Diagram & Sqoop import-export","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/","og_locale":"en_US","og_type":"article","og_title":"Apache Sqoop Architecture - How Sqoop works Internally - DataFlair","og_description":"Sqoop Architecture-Learn Sqoop introduction,complete architecture of sqoop & how sqoop works internally.See Sqoop architecture Diagram & Sqoop import-export","og_url":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-02-06T13:42:34+00:00","article_modified_time":"2021-12-07T07:31:10+00:00","og_image":[{"width":1202,"height":630,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Sqoop-Architecture-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/6e11a88eea98179e861a0933f8959f8f"},"headline":"Apache Sqoop Architecture – How Sqoop works Internally","datePublished":"2018-02-06T13:42:34+00:00","dateModified":"2021-12-07T07:31:10+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/"},"wordCount":645,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Sqoop-Architecture-01.jpg","keywords":["Architecture of sqoop","How Sqoop Works","Sqoop Architecture","Sqoop Architecture and Working","Sqoop Working","working of sqoop"],"articleSection":["Sqoop Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/","url":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/","name":"Apache Sqoop Architecture - How Sqoop works Internally - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Sqoop-Architecture-01.jpg","datePublished":"2018-02-06T13:42:34+00:00","dateModified":"2021-12-07T07:31:10+00:00","description":"Sqoop Architecture-Learn Sqoop introduction,complete architecture of sqoop & how sqoop works internally.See Sqoop architecture Diagram & Sqoop import-export","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Sqoop-Architecture-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Sqoop-Architecture-01.jpg","width":1202,"height":630,"caption":"Introduction to Sqoop Architecture and Working"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/sqoop-architecture-and-working\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Sqoop Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/sqoop\/"},{"@type":"ListItem","position":3,"name":"Apache Sqoop Architecture – How Sqoop works Internally"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/6e11a88eea98179e861a0933f8959f8f","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/54363d8635c383dcd7e3be0ef00dbeef4217c45bf323d27bc2c92366b724df08?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/54363d8635c383dcd7e3be0ef00dbeef4217c45bf323d27bc2c92366b724df08?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/54363d8635c383dcd7e3be0ef00dbeef4217c45bf323d27bc2c92366b724df08?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"At DataFlair, our team of experts is committed to offering high-quality tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. We simplify learning to help you advance your career in the tech world.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam5\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/7369","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=7369"}],"version-history":[{"count":6,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/7369\/revisions"}],"predecessor-version":[{"id":105340,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/7369\/revisions\/105340"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/35226"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=7369"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=7369"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=7369"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}