{"id":7093,"date":"2018-02-02T10:09:38","date_gmt":"2018-02-02T10:09:38","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=7093"},"modified":"2021-12-07T13:20:09","modified_gmt":"2021-12-07T07:50:09","slug":"sqoop-validation","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/sqoop-validation\/","title":{"rendered":"Sqoop Validation – Interfaces & Limitations of Sqoop Validate"},"content":{"rendered":"

Sqoop<\/strong><\/a> Validation is nothing but to validate the data copied. In this article, we will learn the whole concept of validation in Sqoop. After the introduction, we will learn the purpose of validation in sqoop. Moreover, we will also cover sqoop validation interface, Sqoop validation syntax & configuration, examples and limitations\u00a0of sqoop validation as well.<\/span><\/p>\n

Introduction to Sqoop Validation<\/h3>\n

a. Sqoop validation simply means validate the data copied. Basically, either import or Export<\/strong><\/a> by comparing the row counts from the source as well as the target post copy.<\/span>
\nb. Moreover, we use this option to compare the row counts between source as well as the target just after data imported into HDFS<\/strong><\/a>.<\/span>
\nc. While during the imports, all the rows are deleted or added, Sqoop tracks this change. Also updates the log file.<\/span><\/p>\n

\"Sqoop<\/a>

Sqoop Validation – Sqoop Process<\/p><\/div>\n

Interfaces of Sqoop Validation<\/h3>\n

Basically, there are 3 interfaces of Sqoop Validation such as:<\/span><\/p>\n

a. ValidationThreshold<\/h4>\n

We use the ValidationThreshold to determine whether the error margin between the source and target are acceptable: Absolute, Percentage Tolerant and many more. However, the default implementation is AbsoluteValidationThreshold.<\/span>
\nBasically, that ensures that the row counts from source as well as targets are the same.<\/span><\/p>\n

b. ValidationFailureHandler<\/h4>\n

Also, it has once interface with ValidationFailureHandler, that is responsible for handling failures here. Such as log an error\/warning, abort and many more. Although default implementation is LogOnFailureHandler. Here that logs a warning message to the configured logger.<\/span><\/p>\n

c. Validator<\/h4>\n

Basically, by delegating the decision to ValidationThreshold Validator drives the validation logic. Also delegates failure handling to ValidationFailureHandler. Moreover, the default implementation is RowCountValidator here. That validates the row counts from source as well as the target.<\/span><\/p>\n

Purpose to Validate in Sqoop<\/h3>\n

Validation in sqoop’s main purpose is to validate the data copied. Basically,\u00a0 either Sqoop import<\/a> or Export by comparing the row counts from the source as well as the target post copy.<\/span><\/p>\n

\"Sqoop<\/a>

Sqoop Validation – Purpose<\/p><\/div>\n

Syntax of Sqoop Validation<\/h3>\n

$ sqoop import (generic-args) (import-args)
\n$ sqoop export (generic-args) (export-args)
\nBasically, validation arguments are part of import and export arguments.<\/span><\/p>\n

Sqoop Validation Configuration<\/h3>\n

We can say that the validation framework is extensible as well as pluggable in nature. However, it comes with default implementations. Yet we can extend the interfaces to allow custom implementations. Basically, it is possible by passing them as part of the command line arguments as shown below.<\/span><\/p>\n

a. Validator<\/h4>\n\n\n\n\n\n\n
Property<\/strong><\/td>\nValidator<\/strong><\/td>\n<\/tr>\n
Description<\/td>\nIt is a Driver for validation, must implement org.apache.sqoop.validation.Validator<\/span><\/td>\n<\/tr>\n
Supported values<\/td>\nImportant to note that, the value has to be a fully qualified class name<\/span><\/td>\n<\/tr>\n
Default value<\/td>\norg.apache.sqoop.validation.RowCountValidator<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

b. Validation Threshold<\/h4>\n\n\n\n\n\n\n
Property<\/strong><\/td>\nValidation-threshold<\/strong><\/td>\n<\/tr>\n
Description<\/td>\nIt drives the decision on the basis of validation meeting the threshold or not. Must implement\u00a0 \u00a0\u00a0<\/span>\u00a0 org.apache.sqoop.validation.ValidationThreshold<\/span><\/td>\n<\/tr>\n
Supported values<\/td>\nHere, also it is important that the value has to be a fully qualified class name.<\/span><\/td>\n<\/tr>\n
Default value<\/td>\norg.apache.sqoop.validation.AbsoluteValidationThreshold<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

c. Validation Failure Handler<\/h4>\n\n\n\n\n\n\n
Property<\/strong><\/td>\nValidation-failurehandler<\/strong><\/td>\n<\/tr>\n
Description<\/td>\nBasically, it is responsible for handling failures, must implement<\/span>\u00a0org.apache.sqoop.validation.ValidationFailureHandler<\/span><\/td>\n<\/tr>\n
Supported values<\/td>\nLikewise, again it is important that the value has to be a fully qualified class name<\/span><\/td>\n<\/tr>\n
Default value<\/td>\norg.apache.sqoop.validation.AbortOnFailureHandler<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Limitations of Sqoop Validation<\/h3>\n

Since it validates only data copied from a single table into HDFS<\/strong><\/a> currently. So, there are several limitations in the current implementation. Such as:<\/span><\/p>\n