Sqoop Codegen – Arguments & Commands in Codegen with Example

Keeping you updated with latest technology trends, Join DataFlair on Telegram

1. Objective

A tool which generates Java classes that encapsulate and interpret imported records is what we call Sqoop Codegen. However, there is much more to know about Sqoop Codegen in Sqoop. So in this article, we will learn each and every aspect of Codegen. in Sqoop. Apart from the introduction, we will also learn Codegen in Sqoop’s purpose as well as its syntax.

Sqoop Codegen

Introduction to Sqoop Codegen

2. Introduction to Sqoop Codegen and its Purpose

A tool which generates Java classes that encapsulate and interpret imported records is what we call Sqoop-Codegen. Moreover, as part of the Sqoop import process, the Java definition of a record is instantiated. However, we can also perform it separately. Likewise, we can recreate a java source if it gets lost. Also, we can create new versions of a class which uses different delimiters between fields, and so on.
Refer this link to know about Sqoop Export.

If these professionals can make a switch to Big Data, so can you:
Rahul Doddamani Story - DataFlair
Rahul Doddamani
Java → Big Data Consultant, JDA
Follow on
Mritunjay Singh Success Story - DataFlair
Mritunjay Singh
PeopleSoft → Big Data Architect, Hexaware
Follow on
Rahul Doddamani Success Story - DataFlair
Rahul Doddamani
Big Data Consultant, JDA
Follow on
I got placed, scored 100% hike, and transformed my career with DataFlair
Enroll now
Richa Tandon Success Story - DataFlair
Richa Tandon
Support → Big Data Engineer, IBM
Follow on
DataFlair Web Services
You could be next!
Enroll now

3. Codegen Syntax in Sqoop

$ sqoop codegen (generic-args) (codegen-args)
$ sqoop-codegen (generic-args) (codegen-args)
However, the Codegen in sqoop arguments can be entered in any order with respect to one another but the Hadoop generic arguments must precede any Sqoop Codegen arguments only.

4. Sqoop Codegen Arguments

a. Common arguments in Sqoop Codegen

Argument Description
–connect <jdbc-uri> Specify JDBC connect string
–connection-manager <class-name> Specify connection manager class to use
–driver <class-name> Manually specify JDBC driver class to use
–hadoop-mapred-home <dir> Override $HADOOP_MAPRED_HOME
–help Print usage instructions
–password-file Set path for a file containing the authentication password
-P Read password from console
–password <password> Set authentication password
–username <username> Set authentication username
–verbose Print more information while working
–connection-param-file <filename> Optional properties file that provides connection parameters
–relaxed-isolation Set connection transaction isolation to read uncommitted for the mappers.

b. Code generation arguments in Sqoop Codegen

Argument Description
–bindir <dir> Output directory for compiled objects
–class-name <name> Sets the generated class name. This overrides –package-name. When combined with –jar-file, sets the input class.
–jar-file <file> Disable code generation; use specified jar
–outdir <dir> Output directory for generated code
–package-name <name> Put auto-generated classes in this package
–map-column-java <m> Override default mapping from SQL type to Java type for configured columns

c. Output line formatting arguments in Sqoop Codegen

Argument Description
–enclosed-by <char> Sets a required field enclosing character
–escaped-by <char> Sets the escape character
–fields-terminated-by <char> Sets the field separator character
–lines-terminated-by <char> Sets the end-of-line character
–mysql-delimiters Uses MySQL’s default delimiter set: fields: , lines: \n escaped-by: \ optionally-enclosed-by: ‘
–optionally-enclosed-by <char> Sets a field enclosing character

d. Input parsing arguments in Sqoop Codegen

Argument Description
–input-enclosed-by <char> Sets a required field encloser
–input-escaped-by <char> Sets the input escape character
–input-fields-terminated-by <char> Sets the input field separator
–input-lines-terminated-by <char> Sets the input end-of-line character
–input-optionally-enclosed-by <char> Sets a field enclosing character

e. Hive arguments in Sqoop Codegen

Argument Description
–hive-home <dir> Override $HIVE_HOME
–hive-import Import tables into Hive (Uses Hive’s default delimiters if none are set.)
–hive-overwrite Overwrite existing data in the Hive table.
–create-hive-table If set, then the job will fail if the target hive
table exits. By default this property is false.
–hive-table <table-name> Sets the table name to use when importing to Hive.
–hive-drop-import-delims Drops \n, \r, and \01 from string fields when importing to Hive.
–hive-delims-replacement Replace \n, \r, and \01 from string fields with user-defined string when importing to Hive.
–hive-partition-key Name of a hive field to partition is shared on
–hive-partition-value <v> String-value that serves as partition key for this imported into the hive in this job.
–map-column-hive <map> Override default mapping from SQL type to Hive type for configured columns.

Basically, Sqoop generates a file containing the HQL statements to create a table and load data only if Hive arguments are provided to the code generation tool.

5. Sqoop Codegen Commands

All these commands actually define the importance of Codegen tool. Basically, every database table has one DAO class that carry getter and setter methods.
a. Change the directory to /usr/local/hadoop/sbin
$ cd /usr/local/hadoop/sbin
b. Start all hadoop daemons
$ start-all.sh
c. The JPS (Java Virtual Machine Process Status Tool) tool is limited to reporting information on JVMs for which it has the access permissions
$ jps
d. Change the directory to /usr/local/sqoop/bin
$ cd /usr/local/sqoop/bin

6. Codegen Example Invocations in Sqoop

Here, for the employees table of a corporate database we are recreating the record interpretation code.
$ sqoop codegen –connect jdbc:mysql://db.example.com/corp \
   –table employees

7. Conclusion

As a result, we have seen a complete introduction to Sqoop Codegen. Also, we have studied purpose and syntax of codegen in Sqoop. Likewise, we have learned all the arguments and commands associated with Sqoop Codegen. But if you want to ask any query regarding, feel free to ask in the comment section.
Learn about Sqoop eval commands & Sqoop Books
For reference

Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.