{"id":5768,"date":"2018-01-11T11:22:08","date_gmt":"2018-01-11T11:22:08","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=5768"},"modified":"2018-09-18T11:31:58","modified_gmt":"2018-09-18T06:01:58","slug":"apache-hive-vs-spark-sql","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/","title":{"rendered":"Apache Hive vs Spark SQL: Feature wise comparison"},"content":{"rendered":"

1. Objective<\/b><\/h2>\n

While Apache Hive<\/strong><\/a> and Spark SQL<\/strong><\/a> perform the same action, retrieving data, each does the task in a different way. However, Hive is planned as an interface or convenience for querying data stored in HDFS<\/strong><\/a>. Though, MySQL is planned for online operations requiring many reads and writes. So we will discuss Apache Hive vs Spark SQL on the basis of their feature.<\/p>\n

\"Hive<\/a>

Hive vs Spark SQL<\/p><\/div>\n

This blog totally aims at differences between Spark SQL vs Hive in Apache Spark<\/a><\/strong>. We will also cover the features of both individually. To understand more, we will also focus on the usage area of both. Also, there are several limitations with Hive<\/strong><\/a> as well as SQL. We will discuss all in detail to understand the difference between Hive and SparkSQL.<\/p>\n

2. Comparison between Apache Hive vs Spark SQL<\/b><\/h2>\n

At first, we will put light on a brief introduction of each. Afterwards, we will compare both on the basis of various features.<\/p>\n

2.1. Introduction<\/b><\/h3>\n

Apache Hive:<\/strong>
\nApache Hive is built on top of Hadoop. Moreover, It is an open source data warehouse system. Also, helps for analyzing and querying large datasets stored in Hadoop<\/strong><\/a> files. At First, we have to write complex Map-Reduce<\/strong><\/a> jobs. But, using Hive, we just need to submit merely SQL queries. Users who are comfortable with SQL, Hive is mainly targeted towards them.
\nSpark SQL:<\/strong>
\nIn Spark, we use Spark SQL for structured data processing. Moreover, We get more information of the structure of data by using SQL. \u00a0Also, gives information on computations performed. One can achieve extra optimization in Apache Spark, with this extra information. Although, Interaction with Spark SQL is possible in several ways. Such as DataFrame<\/strong><\/a> and the Dataset<\/strong><\/a> API.<\/p>\n

2.2. Initial release<\/b><\/h3>\n

Apache Hive:<\/strong>
\nApache Hive was first released in 2012.
\nSpark SQL: <\/strong>
\nWhile Apache Spark SQL was first released in 2014.<\/p>\n

2.3. Current release<\/b><\/h3>\n

Apache Hive: <\/strong>
\nCurrently released on 24 October 2017: \u00a0version 2.3.1
\nSpark SQL:<\/strong>
\nCurrently released on 09 October 2017: version 2.1.2<\/p>\n

2.4. License<\/b><\/h3>\n

Apache Hive: <\/strong>
\nIt is open sourced, from Apache Version 2.
\nSpark SQL:<\/strong>
\nIt is open sourced, through Apache Version 2.<\/p>\n

2.5. Implementation language<\/b><\/h3>\n

Apache Hive: <\/strong>
\nBasically, we can implement Apache Hive on Java language.
\nSpark SQL:<\/strong>
\nWe can implement Spark SQL on Scala, Java, Python as well as R language.<\/p>\n

2.6. Primary database model<\/b><\/h3>\n

Apache Hive:<\/strong> \u00a0<\/b>
\nPrimarily, its database model is Relational DBMS.
\nSpark SQL:<\/strong>
\nPrimarily, its database model is also Relational DBMS<\/p>\n

2.7. Additional database models<\/b><\/h3>\n

Apache Hive: <\/strong>
\nIt supports an additional database model, i.e. Key-value store
\nSpark SQL: \u00a0<\/strong>
\nAs similar as Hive, it also supports Key-value store as additional database model.<\/p>\n

2.8. Developer<\/strong><\/h3>\n

Apache Hive: <\/strong>
\nHive is originally developed by Facebook. But later donated to the Apache Software Foundation, which has maintained it since.
\nSpark SQL:<\/strong>
\nIt is originally developed by Apache Software Foundation.<\/p>\n

2.9. Server operating systems<\/b><\/h3>\n

Apache Hive:<\/strong>
\nBasically, it supports all Operating Systems with a Java VM.
\nSpark SQL:<\/strong>
\nIt supports several operating systems. For example Linux OS, X, \u00a0and Windows.<\/p>\n

2.10. Data Types<\/b><\/h3>\n

Apache Hive: <\/strong>
\nIt has predefined data types. For example, float or date.
\nSpark SQL:<\/strong>
\nAs similar to Spark SQL, it also has predefined data types. For Example, float or date.<\/p>\n

2.11. Support of SQL<\/strong><\/h3>\n

Apache Hive:<\/strong>
\nIt possesses SQL-like DML and DDL statements.
\nSpark SQL: <\/strong>
\nLike Apache Hive, it also possesses SQL-like DML and DDL statements.<\/p>\n

2.12. APIs and other access methods<\/b><\/h3>\n

Apache Hive:<\/strong>
\nApache Hive supports JDBC, ODBC, and Thrift.
\nSpark SQL:<\/strong>
\nSpark SQL supports only JDBC and ODBC.<\/p>\n

2.13. Programming languages<\/b><\/h3>\n

Apache Hive: <\/strong>
\nWe can use several programming languages in Hive. For example C++, Java, PHP, and Python.
\nSpark SQL: <\/strong>
\nWe can use several programming languages in Spark SQL.\u00a0 For example Java, Python<\/strong><\/a>, R<\/strong><\/a>, and Scala<\/a>. This creates difference between SparkSQL and Hive.<\/p>\n

2.14. Partitioning methods<\/b><\/h3>\n

Apache Hive: <\/strong>
\nIt uses data sharding method for storing data on different nodes.
\nSpark SQL: \u00a0<\/strong>
\nIt uses spark core for storing data on different nodes.<\/p>\n

2.15. Replication methods <\/b><\/h3>\n

Apache Hive:<\/strong>
\nThere is a selectable replication factor for redundantly storing data on multiple nodes.
\nSpark SQL:<\/strong>
\nBasically, for redundantly storing data on multiple nodes, there is a no replication factor in Spark SQL.<\/p>\n

2.16. Concurrency <\/b><\/h3>\n

Apache Hive:<\/strong>
\nBasically, hive supports concurrent manipulation of data.
\nSpark SQL:<\/strong>
\nWhereas, spark SQL also supports concurrent manipulation of data.
\nLet’s see few more difference between Apache Hive vs Spark SQL.<\/p>\n

2.17. Durability<\/strong><\/h3>\n

Apache Hive: <\/strong>
\nBasically, it supports for making data persistent.
\nSpark SQL:<\/strong>
\nAs same as Hive, Spark SQL also support for making data persistent.<\/p>\n

2.18. User concepts<\/strong><\/h3>\n

Apache Hive: <\/strong>
\nThere are access rights for users, groups as well as roles.
\nSpark SQL:<\/strong>
\nThere are no access rights for users.<\/p>\n

2.19. Usage <\/b><\/h3>\n

Apache Hive:<\/strong><\/p>\n

    \n
  • Schema flexibility and evolution.<\/li>\n
  • Also, can portion and bucket, tables in Apache Hive.<\/li>\n
  • As JDBC\/ODBC drivers are available in Hive, we can use it.<\/li>\n<\/ul>\n

    Spark SQL:<\/strong><\/p>\n

      \n
    • Basically, it performs SQL queries.<\/li>\n
    • Through Spark SQL, it is possible to read data from existing Hive installation.<\/li>\n
    • We get the result as Dataset\/DataFrame if we run Spark SQL with another programming language.<\/li>\n<\/ul>\n

      2.20. Limitations <\/b><\/h3>\n

      Apache Hive:<\/strong><\/p>\n

        \n
      • It does not offer real-time queries and row level updates.<\/li>\n
      • Also provides acceptable latency for interactive data browsing.<\/li>\n
      • Hive does not support online transaction processing.<\/li>\n
      • In Apache Hive, latency for queries is generally very high.<\/li>\n<\/ul>\n

        Spark SQL:<\/strong><\/p>\n

          \n
        • It does not support union type<\/li>\n
        • Although, no provision of error for oversize of varchar type<\/li>\n
        • It does not support transactional table<\/li>\n
        • However, no support for Char type<\/li>\n
        • It does not support time-stamp in Avro table.<\/li>\n<\/ul>\n

          3. Conclusion<\/b><\/h2>\n

          Hence, we can not say SparkSQL is not a replacement for Hive neither is the other way. As a result, we have seen that SparkSQL is more spark API and developer friendly. Also, SQL makes programming in spark easier. While, Hive’s ability to switch execution engines, is efficient to query huge data sets. Although, we can just say it\u2019s usage is totally depends on our goals. Apart from it, we have discussed we have discussed Usage as well as limitations above. Also discussed complete discussion of Apache Hive vs Spark SQL. So, hopefully, this blog may answer all the questions occurred in mind regarding Apache Hive vs Spark SQL.<\/p>\n","protected":false},"excerpt":{"rendered":"

          1. Objective While Apache Hive and Spark SQL perform the same action, retrieving data, each does the task in a different way. However, Hive is planned as an interface or convenience for querying data...<\/p>\n","protected":false},"author":6,"featured_media":34564,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[13124,13175],"class_list":["post-5768","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-spark-sql-vs-hive-on-spark","tag-sparksql-vs-hive"],"yoast_head":"\nApache Hive vs Spark SQL: Feature wise comparison - DataFlair<\/title>\n<meta name=\"description\" content=\"Comparison between Apache Hive vs Spark SQL to understand features of both and differences between Spark SQL vs Hive for better understanding.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Hive vs Spark SQL: Feature wise comparison - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Comparison between Apache Hive vs Spark SQL to understand features of both and differences between Spark SQL vs Hive for better understanding.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-11T11:22:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-09-18T06:01:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Apache-Hive-vs-Spark-SQL-Copy-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Hive vs Spark SQL: Feature wise comparison - DataFlair","description":"Comparison between Apache Hive vs Spark SQL to understand features of both and differences between Spark SQL vs Hive for better understanding.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/","og_locale":"en_US","og_type":"article","og_title":"Apache Hive vs Spark SQL: Feature wise comparison - DataFlair","og_description":"Comparison between Apache Hive vs Spark SQL to understand features of both and differences between Spark SQL vs Hive for better understanding.","og_url":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-01-11T11:22:08+00:00","article_modified_time":"2018-09-18T06:01:58+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Apache-Hive-vs-Spark-SQL-Copy-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Apache Hive vs Spark SQL: Feature wise comparison","datePublished":"2018-01-11T11:22:08+00:00","dateModified":"2018-09-18T06:01:58+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/"},"wordCount":1014,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Apache-Hive-vs-Spark-SQL-Copy-01-1.jpg","keywords":["Spark sql vs hive on spark","SparkSQL vs Hive"],"articleSection":["Apache Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/","url":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/","name":"Apache Hive vs Spark SQL: Feature wise comparison - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Apache-Hive-vs-Spark-SQL-Copy-01-1.jpg","datePublished":"2018-01-11T11:22:08+00:00","dateModified":"2018-09-18T06:01:58+00:00","description":"Comparison between Apache Hive vs Spark SQL to understand features of both and differences between Spark SQL vs Hive for better understanding.","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Apache-Hive-vs-Spark-SQL-Copy-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Apache-Hive-vs-Spark-SQL-Copy-01-1.jpg","width":1200,"height":628,"caption":"Apache Hive vs Spark SQL: Feature wise comparison"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/apache-hive-vs-spark-sql\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/spark\/"},{"@type":"ListItem","position":3,"name":"Apache Hive vs Spark SQL: Feature wise comparison"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5768","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=5768"}],"version-history":[{"count":4,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5768\/revisions"}],"predecessor-version":[{"id":34565,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5768\/revisions\/34565"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/34564"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=5768"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=5768"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=5768"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}