{"id":20012,"date":"2018-06-27T14:23:15","date_gmt":"2018-06-27T14:23:15","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=20012"},"modified":"2018-06-27T14:23:15","modified_gmt":"2018-06-27T14:23:15","slug":"hcatalog-loader-and-storer","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/","title":{"rendered":"HCatalog Loader and Storer – Usage & Example"},"content":{"rendered":"

In our last HCatalog tutorial<\/strong>, we discussed HCatalog Commands<\/strong>. Today, we will see HCatalog Loader and Storer. Moreover, we will also discuss the examples of HCatalog loader and storer.<\/span><\/p>\n

Basically, to read and write data in HCatalog-managed tables, the HCatLoader and HCatStorer interfaces are used with Pig scripts.<\/span><\/p>\n

So, let’s learn both\u00a0HCatalog Loader\u00a0and Storer in detail:<\/p>\n

HCatLoader<\/span><\/h2>\n

In order to read data from HCatalog-managed tables, we use HCatLoader with Pig scripts.<\/span><\/p>\n

a. Usage<\/h3>\n

Via a Pig load statement, we can access HCatLoader. <\/span><\/p>\n

A = LOAD 'tablename' USING org.apache.hive.hcatalog.pig.HCatLoader();<\/pre>\n
b. Assumptions<\/h3>\n
However, make sure the table name is specified in single quotes: LOAD ‘tablename’. Also, we must specify our input as ‘dbname.tablename’, if we are using a non-default database. <\/span><\/p>\n
Moreover, we must create our database and table prior to running the Pig script<\/strong>, if we are using Pig 0.9.2 or earlier. In addition, we can issue these create commands in Pig<\/strong> using the SQL command, beginning with Pig 0.10. \u00a0<\/span><\/p>\n
Furthermore, without specifying a database, the Hive <\/strong>metastore lets us create tables; however,\u00a0the database name is ‘default’, if\u00a0we create the tables this way.<\/span><\/p>\n
Although, make sure we can indicate which partitions to scan by immediately following the load statement with a partition filter statement if the table is partitioned.<\/span><\/p>\n
c. HCatLoader Data Types<\/h3>\n
Also, make sure HCatLoader can only read the Hive data types<\/strong>, such as:<\/span><\/p>\n
Types in Hive 0.12.0 and Earlier<\/strong>
\n1.\u00a0boolean
\n2. int
\n3. long
\n4. float<\/span>
\n5. double<\/span>
\n6. string<\/span>
\n7. binary
\nsome complex data types:
\n8. map \u2013 here key type\u00a0must be the string
\n9. ARRAY<any type><\/span>
\n10. struct<any type fields><\/span><\/p>\n
Types in Hive 0.13.0 and Later<\/strong>
\n11. tinyint<\/span>
\n12. smallint
\n13. date
\n14. timestamp<\/span>
\n15. decimal<\/span>
\n16. char(x)<\/span>
\n17. varchar(x)<\/span><\/p>\n
HCatStorer <\/span><\/h2>\n
Further, in order to write data to HCatalog-managed tables, we use HCatStorer with Pig scripts.<\/span><\/p>\n
\"HCatalog<\/a>
HCatalog Loader and Storer<\/p><\/div>\n
a. Usage<\/h3>\n
via a Pig store statement, we access HCatStorer.<\/span><\/p>\n
A = LOAD ...\nB = FOREACH A ...\n...\n...\nmy_processed_data = ...\nSTORE my_processed_data INTO 'tablename'\n  USING org.apache.hive.hcatalog.pig.HCatStorer();<\/pre>\n
b. Assumptions<\/h3>\n
As similar as HCatStorer, here also table name must be in single quotes, like LOAD ‘tablename’. Moreover, to run script make sure that both the database and table must be created prior. Also, we must specify our input as ‘dbname.tablename’, if\u00a0we are using a non-default database. <\/span><\/p>\n
And,\u00a0we need to create our database and table prior to running the Pig script, if\u00a0we are using Pig 0.9.2 or earlier. Further,\u00a0we can issue these create commands in Pig using the SQL command, beginning with Pig 0.10.<\/span><\/p>\n
As the best feature, without even specifying a database, the Hive metastore<\/strong> lets\u00a0us create tables. So, after that, the database name is ‘default’ if\u00a0we create tables this way, and also\u00a0there is no need to specify the database name in the store statement.<\/span><\/p>\n
c. Store Examples<\/h3>\n
By using HCatStorer we can write to a non-partitioned table simply.\u00a0Also, the table contents will be overwritten:<\/span><\/p>\n
store z into 'web_data' using org.apache.hive.hcatalog.pig.HCatStorer();<\/pre>\n
In addition, specify the partition value in the store function, to add one new partition to a partitioned table. Make sure that the whole string should be single-quoted as well as separated with an equals sign:<\/span>
\nstore z into ‘web_data’ using org.apache.hive.hcatalog.pig.HCatStorer(‘datestamp=20110924’);<\/strong>
\nEnsure that the partition column is present in our data,\u00a0then only call HCatStorer with no argument, to write into multiple partitions at once:<\/span><\/p>\n
store z into 'web_data' using org.apache.hive.hcatalog.pig.HCatStorer();\n -- datestamp must be a field in the relation z<\/pre>\n
d. HCatStorer Data Types<\/p>\n
Types in Hive 0.12.0 and Earlier<\/strong>
\n1.\u00a0Boolean
\n2. int<\/span>
\n3. long
\n4. float<\/span>
\n5. double<\/span>
\n6. chararray<\/span>
\n7. bytearray
\nSome complex data types:<\/span>
\n8. map<\/span>
\n9. bag<\/span>
\n10. tuple<\/span><\/p>\n
Types in Hive 0.13.0 and Later<\/strong>
\n11. short
\n12. datetime<\/span>
\n13. bigdecimal<\/span>
\nSo, this was all about HCatalog Loader and Storer. Hope you like our explanation.<\/p>\n
Conclusion: HCatalog Loader and Storer<\/h2>\n
Hence, we have seen the concept of HCatalog Loader and Storer. So, this article will definitely help to clear all doubts regarding HCatalog loader and storer. Still, if any doubt, ask in the comment tab.<\/p>\n","protected":false},"excerpt":{"rendered":"
In our last HCatalog tutorial, we discussed HCatalog Commands. Today, we will see HCatalog Loader and Storer. Moreover, we will also discuss the examples of HCatalog loader and storer. Basically, to read and write...<\/p>\n","protected":false},"author":7,"featured_media":20257,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24],"tags":[5524,5535,5541,5542,5546,5701,13865,15007],"class_list":["post-20012","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hcatalog","tag-hcatalog-loader-storer","tag-hcatalog-tutorial","tag-hcatloader","tag-hcatloader-data-types","tag-hcatstorer","tag-hive-data-types","tag-store-examples","tag-types-in-hive-0-12-0-and-earlier"],"yoast_head":"\nHCatalog Loader and Storer - Usage & Example - DataFlair<\/title>\n<meta name=\"description\" content=\"HCatalog Loader and Storer, HCatLoader, HCatStorer, usage, assumption, HCatalog example, Hcatalog tutorial, what is HCatalog loader and storer\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"HCatalog Loader and Storer - Usage & Example - DataFlair\" \/>\n<meta property=\"og:description\" content=\"HCatalog Loader and Storer, HCatLoader, HCatStorer, usage, assumption, HCatalog example, Hcatalog tutorial, what is HCatalog loader and storer\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-27T14:23:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/HCatalog-Loader-and-Storer-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"HCatalog Loader and Storer - Usage & Example - DataFlair","description":"HCatalog Loader and Storer, HCatLoader, HCatStorer, usage, assumption, HCatalog example, Hcatalog tutorial, what is HCatalog loader and storer","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/","og_locale":"en_US","og_type":"article","og_title":"HCatalog Loader and Storer - Usage & Example - DataFlair","og_description":"HCatalog Loader and Storer, HCatLoader, HCatStorer, usage, assumption, HCatalog example, Hcatalog tutorial, what is HCatalog loader and storer","og_url":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-06-27T14:23:15+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/HCatalog-Loader-and-Storer-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"HCatalog Loader and Storer – Usage & Example","datePublished":"2018-06-27T14:23:15+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/"},"wordCount":628,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/HCatalog-Loader-and-Storer-01.jpg","keywords":["HCatalog Loader & Storer","HCatalog Tutorial","HCatloader","HCatLoader Data Types","HCatStorer","Hive Data Types","Store Examples","Types in Hive 0.12.0 and Earlier"],"articleSection":["HCatalog Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/","url":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/","name":"HCatalog Loader and Storer - Usage & Example - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/HCatalog-Loader-and-Storer-01.jpg","datePublished":"2018-06-27T14:23:15+00:00","description":"HCatalog Loader and Storer, HCatLoader, HCatStorer, usage, assumption, HCatalog example, Hcatalog tutorial, what is HCatalog loader and storer","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/HCatalog-Loader-and-Storer-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/HCatalog-Loader-and-Storer-01.jpg","width":1200,"height":628,"caption":"HCatalog Loader and Storer - Usage & Example"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/hcatalog-loader-and-storer\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"HCatalog Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/hcatalog\/"},{"@type":"ListItem","position":3,"name":"HCatalog Loader and Storer – Usage & Example"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/20012","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=20012"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/20012\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/20257"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=20012"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=20012"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=20012"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}