{"id":14094,"date":"2018-04-27T09:27:19","date_gmt":"2018-04-27T09:27:19","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=14094"},"modified":"2018-04-27T09:27:19","modified_gmt":"2018-04-27T09:27:19","slug":"apache-pig-udf","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/","title":{"rendered":"Pig UDF | Apache Pig User Defined Functions and Its Types"},"content":{"rendered":"

There is an extensive support for User Defined Functions (UDF\u2019s) in Apache Pig<\/strong>. In this article “Apache Pig UDF”, we will learn the whole concept of Apache Pig UDFs. <\/span><\/p>\n

Moreover, we will also learn its introduction. In addition, we will discuss types of Pig UDF, way to write as well as the way to use these UDF\u2019s in detail.<\/span><\/p>\n

What is Apache Pig UDF?<\/span><\/h2>\n

Apache Pig offers extensive support for Pig UDF, in addition to the built-in functions. Basically, we can define our own functions and use them, using these UDF\u2019s. Moreover, in six programming languages, UDF support is available. Such as Java<\/strong>, Jython, Python<\/strong>, JavaScript, Ruby, and Groovy.<\/span><\/p>\n

However, we can say, complete support is only provided in Java. While in all the remaining languages limited support is provided. In addition, we can write UDF\u2019s involving all parts of the processing like data load\/store, column transformation, and aggregation, using Java. <\/span><\/p>\n

Although, make sure the UDF\u2019s written using Java language work efficiently as compared to other languages since Apache Pig has been written in Java.<\/span><\/p>\n

Also, we have a Java repository for UDF\u2019s named Piggybank, in Apache Pig. Basically, we can access Java UDF\u2019s written by other users, and contribute our own UDF\u2019s, using Piggybank.<\/span><\/p>\n

Types of Pig UDF in Java<\/span><\/h2>\n

We can create and use several types of functions while writing Pig UDF using Java, such as:<\/span><\/p>\n

\"Pig<\/a>

Pig UDF – Types of Pig UDF in Java<\/p><\/div>\n

a. Filter Functions <\/span><\/h3>\n

In filter statements, we use the filter functions as conditions. Basically, it accepts a Pig value as input and returns a Boolean value.<\/span><\/p>\n

b. Eval Functions <\/span><\/h3>\n

In FOREACH GENERATE statements, we use the Eval functions. Basically, it accepts a Pig value as input and returns a Pig result.<\/span><\/p>\n

c. Algebraic Functions
\n<\/span><\/h3>\n

In a FOREACH GENERATE statement, we use the Algebraic functions act on inner bags. Basically, to perform full MapReduce operations on an inner bag, we use these functions.<\/span><\/p>\n

Writing Pig UDF using Java<\/span><\/h2>\n

We have to integrate the jar file Pig-0.15.0.jar, in order to write a UDF using Java. However, at first, make sure we have installed Eclipse and Maven in our system because here we are discussing how to write a sample UDF using Eclipse. <\/span>
\nSo, in order to write a UDF function, follow these steps \u2212<\/span><\/p>\n

    \n
  • Firstly, create a new project after opening the Eclipse (say project1).<\/span><\/li>\n
  • Then convert the newly created project into a Maven project.<\/span><\/li>\n
  • Further, copy the following content in the pom.xml. Basically, this file contains the Maven dependencies for Apache Pig and Hadoop\u00a0<\/strong>– core jar files.<\/span><\/li>\n<\/ul>\n
    <project xmlns = \"http:\/\/maven.apache.org\/POM\/4.0.0\"\n  xmlns:xsi = \"http:\/\/www.w3.org\/2001\/XMLSchema-instance\"\n  xsi:schemaLocation = \"http:\/\/maven.apache.org\/POM\/4.0.0http:\/\/maven.apache .org\/xsd\/maven-4.0.0.xsd\">\n  <modelVersion>4.0.0<\/modelVersion>\n  <groupId>Pig_Udf<\/groupId>\n  <artifactId>Pig_Udf<\/artifactId>\n  <version>0.0.1-SNAPSHOT<\/version>\n  <build>\n     <sourceDirectory>src<\/sourceDirectory>\n     <plugins>\n        <plugin>\n           <artifactId>maven-compiler-plugin<\/artifactId>\n           <version>3.3<\/version>\n           <configuration>\n              <source>1.7<\/source>\n              <target>1.7<\/target>\n           <\/configuration>\n        <\/plugin>\n     <\/plugins>\n  <\/build>\n  <dependencies>\n     <dependency>\n        <groupId>org.apache.pig<\/groupId>\n        <artifactId>pig<\/artifactId>\n        <version>0.15.0<\/version>\n     <\/dependency>\n     <dependency>\n        <groupId>org.apache.hadoop<\/groupId>\n        <artifactId>hadoop-core<\/artifactId>\n        <version>0.20.2<\/version>\n     <\/dependency>\n  <\/dependencies>\n<\/project><\/pre>\n
      \n
    • Afterward, save the file and refresh it. We can find the downloaded jar files, in the Maven dependencies section.<\/span><\/li>\n
    • Then create a new class file with name Sample_Eval. Also, copy the following content in it.<\/span><\/li>\n<\/ul>\n
      import java.io.IOException;\nimport org.apache.pig.EvalFunc;\nimport org.apache.pig.data.Tuple;\nimport java.io.IOException;\nimport org.apache.pig.EvalFunc;\nimport org.apache.pig.data.Tuple;\npublic class Sample_Eval extends EvalFunc<String>{\n  public String exec(Tuple input) throws IOException {\n     if (input == null || input.size() == 0)\n     return null;\n     String str = (String)input.get(0);\n     return str.toUpperCase();\n  }\n}<\/pre>\n
      However, it is necessary to inherit the EvalFunc class and provide implementation to exec() function, while writing UDF\u2019s. The code required for the UDF is written, within this function. Also, see that we have to return the code to convert the contents of the given column to uppercase, in the above example.<\/span><\/p>\n
        \n
      • Right-click on the Sample_Eval.java file just after compiling the class without errors that display us a menu, then select Export.<\/span><\/li>\n
      • Now, we will get the following window, by clicking Export. Then, click on the JAR file.<\/span><\/li>\n
      • Also, by clicking Next> button proceed further. In this way, we will get another window. Through that, we need to enter the path in the local file system. Especially, where we need to store the jar file.<\/span><\/li>\n
      • Now click the Finish button, we can see, a Jar file sample_udf.jar is created, in the specified folder. That jar file contains the UDF written in Java.<\/span><\/li>\n<\/ul>\n
        Using Pig UDF<\/span><\/h2>\n
        Now, follow these steps, after writing the UDF and generating the Jar file \u2212<\/span><\/p>\n
        Step 1: Registering the Jar file<\/strong>
        \nBasically, using the Register operator, we have to register the Jar file that contains the UDF, just after writing UDF (in Java). Also, users can intimate the location of the UDF to Apache Pig, by registering the Jar file.<\/span><\/p>\n
          \n
        • Syntax<\/strong><\/li>\n<\/ul>\n
          So, the syntax of the Register operator is-<\/span><\/p>\n
          REGISTER path;<\/pre>\n
            \n
          • Example<\/strong><\/li>\n<\/ul>\n
            For Example, let\u2019s register the sample_udf.jar created above.<\/span><\/p>\n
            Start Apache Pig in local mode and register the jar file sample_udf.jar as shown below.\n$cd PIG_HOME\/bin\n$.\/pig \u2013x local\nREGISTER '\/$PIG_HOME\/sample_udf.jar'<\/pre>\n
            It is very important to suppose the Jar file in the path \u2212 \/$PIG_HOME\/sample_udf.jar<\/span><\/p>\n
            Step 2: Defining Alias<\/strong>
            \nUsing the Define operator, we can define an alias to UDF after registering the UDF.<\/span><\/p>\n
              \n
            • Syntax<\/strong><\/li>\n<\/ul>\n
              So, the syntax of the Define operator.<\/span><\/p>\n
              DEFINE alias {function | [`command` [input] [output] [ship] [cache] [stderr] ] };<\/pre>\n
                \n
              • Example<\/strong><\/li>\n<\/ul>\n
                Let\u2019s define the alias for sample_eval.<\/span><\/p>\n
                DEFINE sample_eval sample_eval();<\/pre>\n
                Step 3: Using the UDF<\/strong>
                \nWe can use the UDF same as the built-in functions, after defining the alias. Then assume there is a file named Stu_data in the HDFS \/Pig_Data\/ directory. It’s content is:<\/span><\/p>\n
                001,Rohit,22,new york\n002,Sohail,23,Kolkata\n003,Ankur,23,Tokyo\n004,Vishal,25,London\n005,Dheeraj,23,Bhubaneswar\n006,Mitali,22,Chennai\n007,Raj,22,new york\n008,Somesh,23,Kolkata\n009,Mehul,25,Tokyo\n010,Shreyash,25,London\n011,Shan,25,Bhubaneswar\n012,Kajal,22,Chennai<\/pre>\n
                Also, suppose we have loaded this file into Pig.<\/span><\/p>\n
                grunt> Stu_data = LOAD 'hdfs:\/\/localhost:9000\/pig_data\/Stu1.txt' USING PigStorage(',')\n  as (id:int, name:chararray, age:int, city:chararray);<\/pre>\n
                Further, let\u2019s convert the names of the students into the upper case using the UDF sample_eval.<\/span><\/p>\n
                grunt> Upper_case = FOREACH Stu_data GENERATE sample_eval(name);<\/pre>\n
                Afterward, verify the contents of the relation Upper_case.<\/span><\/p>\n
                grunt> Dump Upper_case;\n(Rohit)\n(Sohail)\n(Ankur)\n(Vishal)\n(Dheeraj)\n(Mitali)\n(Raj)\n(Somesh)\n(Mehul)\n(Shreyash)\n(Shan)\n(Kajal)<\/pre>\n
                So, this was all about Apache Pig User Defined Functions. Hope you like our explanation.<\/p>\n
                Conclusion<\/span><\/h2>\n
                As a result, we have seen the whole concept of Apache Pig User Defined Functions (UDF\u2019s). In addition, we discussed types of Pig UDF, Writing Pig UDF with Java. At last, we have learned how to use the Pig UDF. Still, if you want to ask any questions, please ask through the comment section.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
                There is an extensive support for User Defined Functions (UDF\u2019s) in Apache Pig. In this article “Apache Pig UDF”, we will learn the whole concept of Apache Pig UDFs. Moreover, we will also learn...<\/p>\n","protected":false},"author":7,"featured_media":14654,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[40],"tags":[863,889,890,16667,16668,15089,16300],"class_list":["post-14094","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pig","tag-apache-pig","tag-apache-pig-udf","tag-apache-pig-user-defined-function","tag-pig-udf","tag-pig-user-defined-functions","tag-types-of-pig-udf-in-java","tag-writing-pig-udf-using-java"],"yoast_head":"\nPig UDF | Apache Pig User Defined Functions and Its Types - DataFlair<\/title>\n<meta name=\"description\" content=\"Apache Pig UDF: what is Apache Pig UDF, i.e. Apache Pig User Defined Function, Types of Pig UDF in Java, Writing Pig UDF using Java with Syntax and example\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Pig UDF | Apache Pig User Defined Functions and Its Types - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Apache Pig UDF: what is Apache Pig UDF, i.e. Apache Pig User Defined Function, Types of Pig UDF in Java, Writing Pig UDF using Java with Syntax and example\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-04-27T09:27:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Pig-User-Defined-Functions-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Pig UDF | Apache Pig User Defined Functions and Its Types - DataFlair","description":"Apache Pig UDF: what is Apache Pig UDF, i.e. Apache Pig User Defined Function, Types of Pig UDF in Java, Writing Pig UDF using Java with Syntax and example","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/","og_locale":"en_US","og_type":"article","og_title":"Pig UDF | Apache Pig User Defined Functions and Its Types - DataFlair","og_description":"Apache Pig UDF: what is Apache Pig UDF, i.e. Apache Pig User Defined Function, Types of Pig UDF in Java, Writing Pig UDF using Java with Syntax and example","og_url":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-04-27T09:27:19+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Pig-User-Defined-Functions-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"Pig UDF | Apache Pig User Defined Functions and Its Types","datePublished":"2018-04-27T09:27:19+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/"},"wordCount":922,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Pig-User-Defined-Functions-01-1.jpg","keywords":["apache pig","Apache Pig UDF","Apache Pig User Defined Function","Pig UDF","Pig User defined functions","Types of Pig UDF in Java","Writing Pig UDF using Java"],"articleSection":["Pig Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/","url":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/","name":"Pig UDF | Apache Pig User Defined Functions and Its Types - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Pig-User-Defined-Functions-01-1.jpg","datePublished":"2018-04-27T09:27:19+00:00","description":"Apache Pig UDF: what is Apache Pig UDF, i.e. Apache Pig User Defined Function, Types of Pig UDF in Java, Writing Pig UDF using Java with Syntax and example","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/apache-pig-udf\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Pig-User-Defined-Functions-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Pig-User-Defined-Functions-01-1.jpg","width":1200,"height":628,"caption":"Pig UDF | Apache Pig User Defined Functions"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/apache-pig-udf\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Pig Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/pig\/"},{"@type":"ListItem","position":3,"name":"Pig UDF | Apache Pig User Defined Functions and Its Types"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14094","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=14094"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14094\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/14654"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=14094"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=14094"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=14094"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}