{"id":5847,"date":"2018-01-12T06:32:44","date_gmt":"2018-01-12T06:32:44","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=5847"},"modified":"2018-09-18T11:26:56","modified_gmt":"2018-09-18T05:56:56","slug":"spark-sql-features","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/","title":{"rendered":"Spark SQL Features You must know"},"content":{"rendered":"<h2><b>1. Objective<\/b><\/h2>\n<p>In this document, we will see various shining <a href=\"https:\/\/data-flair.training\/blogs\/spark-sql-tutorial\/\"><strong>Spark SQL<\/strong><\/a> features. There are many features Like Unified Data Access, High Compatibility and many more. We will focus on each feature in detail. But, before learning features of Spark SQL, we will also study brief introduction to Spark SQL.<\/p>\n<div id=\"attachment_5852\" style=\"width: 1210px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-5852\" class=\"wp-image-5852 size-full\" src=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01.jpg\" alt=\"Spark SQL Features\" width=\"1200\" height=\"628\" srcset=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01.jpg 1200w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-150x79.jpg 150w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-300x157.jpg 300w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-768x402.jpg 768w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-1024x536.jpg 1024w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/a><p id=\"caption-attachment-5852\" class=\"wp-caption-text\">Spark SQL Features<\/p><\/div>\n<h2><b>2. Introduction to Spark SQL<\/b><\/h2>\n<p>In <a href=\"https:\/\/data-flair.training\/blogs\/apache-spark-for-beginners\/\"><strong>Apache Spark<\/strong><\/a>, Spark SQL is a module for working with structured data. Spark SQL supports distributed <a href=\"https:\/\/data-flair.training\/blogs\/apache-spark-in-memory-computing\/\"><strong>in-memory computations<\/strong><\/a> on a huge scale. It divulges the information about the structure of both computations as well as data. To perform extra optimizations, this extra information turns very helpful. We can easily execute SQL queries through it.<br \/>\nIn addition, to read data from an existing <a href=\"https:\/\/data-flair.training\/blogs\/apache-hive-installation-tutorial\/\"><strong>Hive installation<\/strong><\/a>, we can use Spark SQL. The results come as <a href=\"https:\/\/data-flair.training\/blogs\/apache-spark-dataset-tutorial\/\"><strong>Dataset<\/strong><\/a>\/<a href=\"https:\/\/data-flair.training\/blogs\/apache-spark-sql-dataframe-tutorial\/\"><strong>DataFrame<\/strong><\/a> When SQL run in another programming language. We can interact with the SQL interface, by using the command-line or over JDBC\/ODBC.<br \/>\nThe 3 main capabilities of using structured and semi-structured data, by Spark SQL. Such as:<\/p>\n<ul>\n<li>It grants a DataFrame abstraction in Scala, Java, as well as <a href=\"https:\/\/data-flair.training\/blogs\/python-tutorial-for-beginners\/\"><strong>Python<\/strong><\/a>. Also, \u00a0simplifies working with structured datasets. Here, DataFrames are similar to tables in a relational database.<\/li>\n<li>In various structured formats, Spark SQL can read and write data. For Example Hive Tables, JSON and Parquet.<\/li>\n<li>\u00a0We can query the data by using Spark SQL. Both inside a Spark program and from external tools that connect to Spark SQL.<\/li>\n<\/ul>\n<p>In Spark SQL, developers can switch back and forth between different APIs, as same as Spark. Therefore, it bestows the most natural way to express the given <a href=\"https:\/\/data-flair.training\/blogs\/spark-rdd-operations-transformations-actions\/\"><strong>Transformations<\/strong>.<\/a><\/p>\n<h2><b>3. Spark SQL features<br \/>\n<\/b><\/h2>\n<h3><b>a. Integrated<\/b><\/h3>\n<p>Integrate is simply defined as combining or merge. Here, Spark SQL queries are integrated with Spark programs. Through Spark SQL we are allowed to query structured data inside Spark programs. This is possible by using SQL or a DataFrame that can be used in Java, <a href=\"https:\/\/data-flair.training\/blogs\/why-you-should-learn-scala-introductory-tutorial\/\">Scala<\/a>.<br \/>\nWe can run streaming computation through it. Developers write a batch computation against the DataFrame \/ Dataset API to run it. After that to run it in a streaming fashion Spark itself increments the computation. Developers leverage the advantage of it that they don\u2019t have to manage state, failures on own. Even no need keep the application in sync with batch jobs. Despite, the streaming job always gives the same answer as a batch job on the same data.<\/p>\n<h3><b>b. Unified Data Access<\/b><\/h3>\n<p>To access a variety of data sources DataFrames and SQL support a common way. Data Sources like Hive, Avro, Parquet, ORC, JSON, as well as JDBC. It helps to join the data from these sources. To accommodate all the existing users into Spark SQL, it turns out to be very helpful.<\/p>\n<h3><b>c. High compatibility<\/b><\/h3>\n<p>We are allowed to run unmodified Hive queries on existing warehouses in Spark SQL. With existing Hive data, queries and UDFs, Spark SQL offers full compatibility. Also, rewrites the MetaStore and Hive frontend.<\/p>\n<h3><b>d. Standard Connectivity<\/b><\/h3>\n<p>We can easily connect Spark SQL through JDBC or ODBC.\u00a0 For connectivity for business intelligence tools, Both turned as industry norms. Also, includes industry standard JDBC and ODBC connectivity with server mode.<\/p>\n<h3><b>e. Scalability<\/b><\/h3>\n<p>It takes advantage of RDD model, to support large jobs and mid-query <a href=\"https:\/\/data-flair.training\/blogs\/fault-tolerance-in-apache-spark\/\">fault tolerance<\/a>. For interactive as well as long queries, it uses the same engine.<\/p>\n<h3><b>f. Performance Optimization<\/b><\/h3>\n<p>In Spark SQL, query optimization engine converts each SQL query into a logical plan. Afterwards, it converts to many physical execution plans. \u00a0At the time of execution, it selects the most optimal physical plan, among the entire plan. It ensures fast execution of HIVE queries.<\/p>\n<h3><b>g. For \u00a0batch processing of Hive tables<\/b><\/h3>\n<p>While working with Hive tables, we can use Spark SQL for Batch Processing in them.<\/p>\n<h2><b>4. Conclusion<\/b><\/h2>\n<p>Hence, we have seen all Spark SQL features in detail. As a result, we have learned, \u00a0Spark SQL is a module of Spark that analyses structured data. Basically, it offers scalability and ensures high compatibility of the system. Also, allow standard connectivity through JDBC or ODBC. Therefore, it bestows the most natural way to express the structured data. Moreover, it enhances its working efficiency with above-mentioned Spark SQL features.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>1. Objective In this document, we will see various shining Spark SQL features. There are many features Like Unified Data Access, High Compatibility and many more. We will focus on each feature in detail.&#46;&#46;&#46;<\/p>\n","protected":false},"author":6,"featured_media":34550,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[4636,4637,13118,13174],"class_list":["post-5847","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-features-of-spark-sql","tag-features-of-sparksql","tag-spark-sql-features","tag-sparksql-features"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Spark SQL Features You must know - DataFlair<\/title>\n<meta name=\"description\" content=\"Spark SQL features - Introduction to SparkSQl,features of Spark SQL like high compatibility, scalability, integration, performance optimization, data access\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/spark-sql-features\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Spark SQL Features You must know - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Spark SQL features - Introduction to SparkSQl,features of Spark SQL like high compatibility, scalability, integration, performance optimization, data access\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/spark-sql-features\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-12T06:32:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-09-18T05:56:56+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Spark SQL Features You must know - DataFlair","description":"Spark SQL features - Introduction to SparkSQl,features of Spark SQL like high compatibility, scalability, integration, performance optimization, data access","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/","og_locale":"en_US","og_type":"article","og_title":"Spark SQL Features You must know - DataFlair","og_description":"Spark SQL features - Introduction to SparkSQl,features of Spark SQL like high compatibility, scalability, integration, performance optimization, data access","og_url":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-01-12T06:32:44+00:00","article_modified_time":"2018-09-18T05:56:56+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Spark SQL Features You must know","datePublished":"2018-01-12T06:32:44+00:00","dateModified":"2018-09-18T05:56:56+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/"},"wordCount":702,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-1.jpg","keywords":["Features of Spark SQL","Features of SparkSQL","spark sql features","SparkSQL features"],"articleSection":["Apache Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/spark-sql-features\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/","url":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/","name":"Spark SQL Features You must know - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-1.jpg","datePublished":"2018-01-12T06:32:44+00:00","dateModified":"2018-09-18T05:56:56+00:00","description":"Spark SQL features - Introduction to SparkSQl,features of Spark SQL like high compatibility, scalability, integration, performance optimization, data access","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/spark-sql-features\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Features-of-Spark-SQL-01-1.jpg","width":1200,"height":628,"caption":"Spark SQL Features"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/spark-sql-features\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/spark\/"},{"@type":"ListItem","position":3,"name":"Spark SQL Features You must know"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5847","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=5847"}],"version-history":[{"count":4,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5847\/revisions"}],"predecessor-version":[{"id":34551,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5847\/revisions\/34551"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/34550"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=5847"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=5847"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=5847"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}