{"id":10778,"date":"2018-03-15T00:00:32","date_gmt":"2018-03-15T00:00:32","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=10778"},"modified":"2018-03-15T00:00:32","modified_gmt":"2018-03-15T00:00:32","slug":"skew-join-in-hive","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/","title":{"rendered":"Skew Join in Hive – Working, Tips & Examples"},"content":{"rendered":"

In our last article, we discuss Bucket Map Join in<\/strong> Hive<\/strong>, today we discuss Skew Join in hive. In Apache Hive<\/strong>, when there is a table with skew data in the joining column, we use Skew join in Hive. <\/span><\/p>\n

However, there are much more to know about Skew join feature in Apache Hive. So, in this article, we will learn the what is Skew Join in Hive. Also, it includes Parameter of Hive Skew Join, limitations of Skew Join and Skew Join in Hive Examples.<\/span><\/p>\n

Skew join in Hive<\/span><\/h2>\n

Basically, when there is a table with skew data in the joining column, we use skew join feature. On defining what is skewed table, it is a table that is having values that are present in large numbers in the table compared to other data.<\/span><\/p>\n

However, while the rest of the data is stored in a separate file Skew data is stored in a separate file.<\/span><\/p>\n

\"Skew<\/a>

Skew Join<\/p><\/div>\n

a. Parameter<\/h3>\n

However, to be set for a Hive skew join we need the following parameter:<\/span>
\nset <\/span>
\n<\/span>hive.optimize.skewjoin=true;<\/span>
\n<\/span>set hive.skewjoin.key=100000;<\/span><\/p>\n

b. Command to use<\/h3>\n

Moreover, a bucket sort merge map Join in Hive, Run the following command:<\/span>
\n<\/span>
\n<\/span>SELECT a.* FROM Sales a JOIN Sales_orc b ON a.id = b.id;<\/span><\/p>\n

How Hive Skew Join Works<\/span><\/h2>\n

However, let\u2019s assume if table A join B, and A has skew data “1” in joining column.<\/span>
\n<\/span>At First store, the rows with key 1 in an in-memory hash table and read B.<\/span> Further to read A \u00a0run a set of mappers. Afterward, do the following:<\/p>\n

    \n
  • Make sure use the hashed version of B to compute the result since it has key 1.<\/span><\/li>\n
  • Then, send all other keys to a reducer which does the join. Basically, from a mapper, this reducer will get rows of B also.<\/span><\/li>\n<\/ul>\n

    Hence, as a result, we end up reading only B twice. Basically, that implies that the skewed keys in A are only read and processed by the Mapper. Also, they are not sent to the reducer. Moreover, remaining keys in A go through only a single Map\/Reduce.<\/span>
    \n<\/span><\/p>\n

    However, the assumption is that B has few rows with keys which are skewed in A.<\/span> Hence, in this way these rows can be loaded into the memory.\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0<\/span><\/p>\n

    Skew Join – Use Case<\/span><\/h2>\n
      \n
    • Basically, on the joining column, one table has huge skew values.<\/span><\/li>\n<\/ul>\n

      Disadvantages of Skew Join in Hive<\/span><\/h2>\n

      Here, are some Limitations of Hive Skew Join are discussed:<\/p>\n

        \n
      • So, the major disadvantage of it is One table is read twice here.<\/span><\/li>\n
      • Moreover, it is necessary that users should be aware of the skew key.<\/span><\/li>\n<\/ul>\n

        Skew Join in Hive\u00a0<\/span>Example<\/span><\/h2>\n

        Let’s discuss Hive Join example in brief.
        \nhive> explain select a.* from passwords a, passwords2 b where a.col0=b.col1; \u00a0<\/span>
        \n<\/span>OK<\/span>
        \n<\/span>STAGE DEPENDENCIES: \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>
        \n<\/span> \u00a0Stage-7 is a root stage , consists of Stage-1<\/span>
        \n<\/span> \u00a0Stage-1<\/span>
        \n<\/span> \u00a0Stage-4 depends on stages: Stage-1 , consists of Stage-8<\/span>
        \n<\/span> \u00a0Stage-8<\/span>
        \n<\/span> \u00a0Stage-3 depends on stages: Stage-8<\/span>
        \n<\/span> \u00a0Stage-0 is a root stage<\/span>
        \n<\/span>
        \n<\/span>STAGE PLANS:<\/span>
        \n<\/span> \u00a0Stage: Stage-7<\/span>
        \n<\/span> \u00a0\u00a0\u00a0Conditional Operator<\/span>
        \n<\/span>
        \n<\/span> \u00a0Stage: Stage-1<\/span>
        \n<\/span> \u00a0\u00a0\u00a0Map Reduce<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0Map Operator Tree:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0TableScan<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0alias: b<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 9961472 Data size: 477102080 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Reduce Output Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0key expressions: col1 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0sort order: +<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Map-reduce partition columns: col1 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 9961472 Data size: 477102080 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0value expressions: col1 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0TableScan<\/span>
        \n
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0alias: a<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 9963904 Data size: 477218560 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Reduce Output Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0key expressions: col0 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0sort order: +<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Map-reduce partition columns: col0 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 9963904 Data size: 477218560 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0value expressions: col0 (type: string), col1 (type: string), col2 (type: string), col3 (type: string), col4 (type: string), col5 (type: string), col6 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0Reduce Operator Tree:<\/span>
        \n
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Join Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0condition map:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Inner Join 0 to 1<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0condition expressions:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a00 {VALUE._col0} {VALUE._col1} {VALUE._col2} {VALUE._col3} {VALUE._col4} {VALUE._col5} {VALUE._col6}<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01 {VALUE._col1}<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0handleSkewJoin: true<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col10<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 10960295 Data size: 524940416 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Filter Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0predicate: (_col0 = _col10) (type: boolean)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 5480147 Data size: 262470184 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Select Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 5480147 Data size: 262470184 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0File Output Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0compressed: false<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Statistics: Num rows: 5480147 Data size: 262470184 Basic stats: COMPLETE Column stats: NONE<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0table:<\/span>
        \n
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0input format: org.apache.hadoop.mapred.TextInputFormat<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe<\/span>
        \n<\/span>
        \n<\/span> \u00a0Stage: Stage-4<\/span>
        \n<\/span> \u00a0\u00a0\u00a0Conditional Operator<\/span>
        \n<\/span>
        \n<\/span> \u00a0Stage: Stage-8<\/span>
        \n<\/span> \u00a0\u00a0\u00a0Map Reduce Local Work<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0Alias -> Map Local Tables:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01 <\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Fetch Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0limit: -1<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0Alias -> Map Local Operator Tree:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01 <\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0TableScan<\/span>
        \n
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0HashTable Sink Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0condition expressions:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a00 {0_VALUE_0} {0_VALUE_1} {0_VALUE_2} {0_VALUE_3} {0_VALUE_4} {0_VALUE_5} {0_VALUE_6}<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01 {1_VALUE_0}<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0keys:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a00 joinkey0 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01 joinkey0 (type: string)<\/span>
        \n<\/span>
        \n<\/span> \u00a0Stage: Stage-3<\/span>
        \n<\/span> \u00a0\u00a0\u00a0Map Reduce<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0Map Operator Tree:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0TableScan<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Map Join Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0condition map:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Inner Join 0 to 1<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0condition expressions:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a00 {0_VALUE_0} {0_VALUE_1} {0_VALUE_2} {0_VALUE_3} {0_VALUE_4} {0_VALUE_5} {0_VALUE_6}<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01 {1_VALUE_0}<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0keys:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a00 joinkey0 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01 joinkey0 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col10<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Filter Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0predicate: (_col0 = _col10) (type: boolean)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Select Operator<\/span>
        \n
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0File Output Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0compressed: false<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0table:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0input format: org.apache.hadoop.mapred.TextInputFormat<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0Local Work:<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Map Reduce Local Work<\/span>
        \n<\/span>
        \n<\/span> \u00a0Stage: Stage-0<\/span>
        \n<\/span> \u00a0\u00a0\u00a0Fetch Operator<\/span>
        \n<\/span> \u00a0\u00a0\u00a0\u00a0\u00a0limit: -1<\/span>
        \n<\/span>
        \n<\/span>Time taken: 0.331 seconds, Fetched: 110 row(s)<\/span>
        \n
        \n<\/span>Hence, we can see that there are 2 join operators.<\/span> Such as, one of them is common join and the other one is map join.
        \nSo, it shows “handleSkewJoin: true”.<\/span><\/p>\n

        Tips For Skew Join in Hive<\/span><\/h2>\n

        Following are some Hive Skew Join Tips:<\/p>\n

          \n
        1. However, to be set to enable skew join, we require the below parameter.<\/span>
          \n<\/span>set hive.optimize.skewjoin=true;<\/span><\/li>\n
        2. Moreover, since if we get a skew key in join here it the parameter below that determine.<\/span>
          \n<\/span>Also, we think the key as a skew join key since we see more than the specified number of rows with the same key in join operator.<\/span>
          \n<\/span>set hive.skewjoin.key=100000;<\/span><\/li>\n<\/ol>\n

          Conclusion<\/span><\/h2>\n

          As a result, we have seen all the concept of\u00a0<\/span>skew join feature in Apache Hive. We study what is Skew join in Hive, Parameter of Hive Skew Join, limitations of Skew Join, Skew Join in Hive Examples, Hive Skew Join Working, Skew Join – Use Case, and Tips For Skew Join in Hive. <\/span><\/p>\n

          Still, if any doubt occurs, please share through the comment section. Though, we assure we will get back to you definitely.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

          In our last article, we discuss Bucket Map Join in Hive, today we discuss Skew Join in hive. In Apache Hive, when there is a table with skew data in the joining column, we...<\/p>\n","protected":false},"author":7,"featured_media":10813,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[26],"tags":[3961,5773,5787,5788,8269,8520,9403,12936,12937,12938,12939,12940,15192,15952,15953],"class_list":["post-10778","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hive","tag-disadvantages-of-skew-join-in-hive","tag-hive-reducer-skew","tag-hive-skew-group-by","tag-hive-skewed-table-example","tag-limitations-of-skew-join","tag-map-join-in-hive","tag-parameter-of-hive-skew-join","tag-skew-join","tag-skew-join-in-hive","tag-skew-join-in-hive-example","tag-skew-join-optimization-spark","tag-skew-join-spark","tag-use-case-of-skew-join","tag-what-is-skew-join-in-hive","tag-what-is-skewed-table"],"yoast_head":"\nSkew Join in Hive - Working, Tips & Examples - DataFlair<\/title>\n<meta name=\"description\" content=\"What is Skew join in Hive, Parameter & limitations of Skew Join in hive, Skew Join in Hive Example, Working of Skew Join, Use Case & Tips For Hive Skew Join\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Skew Join in Hive - Working, Tips & Examples - DataFlair\" \/>\n<meta property=\"og:description\" content=\"What is Skew join in Hive, Parameter & limitations of Skew Join in hive, Skew Join in Hive Example, Working of Skew Join, Use Case & Tips For Hive Skew Join\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-03-15T00:00:32+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Skew-join-in-Apache-Hive-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Skew Join in Hive - Working, Tips & Examples - DataFlair","description":"What is Skew join in Hive, Parameter & limitations of Skew Join in hive, Skew Join in Hive Example, Working of Skew Join, Use Case & Tips For Hive Skew Join","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/","og_locale":"en_US","og_type":"article","og_title":"Skew Join in Hive - Working, Tips & Examples - DataFlair","og_description":"What is Skew join in Hive, Parameter & limitations of Skew Join in hive, Skew Join in Hive Example, Working of Skew Join, Use Case & Tips For Hive Skew Join","og_url":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-03-15T00:00:32+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Skew-join-in-Apache-Hive-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"Skew Join in Hive – Working, Tips & Examples","datePublished":"2018-03-15T00:00:32+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/"},"wordCount":1148,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Skew-join-in-Apache-Hive-01.jpg","keywords":["Disadvantages of Skew Join in Hive","hive reducer skew","hive skew group by","hive skewed table example","limitations of Skew Join","Map Join in Hive","Parameter of Hive Skew Join","skew join","Skew join in hive","Skew Join in Hive Example","skew join optimization spark","skew join spark","Use Case of Skew Join","what is Skew join in Hive","what is skewed table"],"articleSection":["Hive Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/","url":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/","name":"Skew Join in Hive - Working, Tips & Examples - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Skew-join-in-Apache-Hive-01.jpg","datePublished":"2018-03-15T00:00:32+00:00","description":"What is Skew join in Hive, Parameter & limitations of Skew Join in hive, Skew Join in Hive Example, Working of Skew Join, Use Case & Tips For Hive Skew Join","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Skew-join-in-Apache-Hive-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Skew-join-in-Apache-Hive-01.jpg","width":1200,"height":628,"caption":"Skew Join in Hive"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/skew-join-in-hive\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Hive Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/hive\/"},{"@type":"ListItem","position":3,"name":"Skew Join in Hive – Working, Tips & Examples"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/10778","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=10778"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/10778\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/10813"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=10778"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=10778"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=10778"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}