{"id":758,"date":"2016-08-02T10:22:26","date_gmt":"2016-08-02T10:22:26","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=758"},"modified":"2018-11-19T16:04:57","modified_gmt":"2018-11-19T10:34:57","slug":"apache-spark-sql","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-spark-sql\/","title":{"rendered":"Apache Spark SQL Tutorial – Quick Introduction Guide"},"content":{"rendered":"

1. Objective – Spark SQL<\/h2>\n

In this Apache Spark SQL tutorial, we will understand various components and terminologies of Spark<\/a> SQL like what is DataSet and DataFrame, what is SqlContext<\/strong> and HiveContext<\/strong> and What are the features of Spark SQL?<\/p>\n

After understanding\u00a0What is Apache Spark<\/a>, in this tutorial we will discuss about Apache Spark SQL. Spark SQL is Spark module for structured data processing. It runs on top of Spark Core.\u00a0It offers much tighter integration between relational and procedural processing, through declarative DataFrame<\/strong> and Datasets API<\/strong>. These are the ways which enable users to run SQL queries over Spark.<\/p>\n

So, let’s start Spark SQL Tutorial.<\/p>\n

\"Apache<\/a>

Apache Spark SQL Tutorial – Quick Introduction Guide<\/p><\/div>\n

2. What is Apache Spark SQL?<\/h2>\n

Apache Spark SQL<\/strong> integrates relational processing with Sparks functional programming. It is Spark module for structured data processing. Spark SQL blurs the line between RDD<\/a><\/strong> and relational table. It also offers much tighter integration between relational and procedural processing, through declarative DataFrame APIs which integrates with Spark code. It thus provides higher optimization. DataFrame API<\/strong> and Datasets API<\/strong> are the ways to interact with Spark SQL. As a result, with Spark SQL, Apache Spark is accessible to more users and improves optimization for current ones.<\/p>\n

Spark SQL runs on top of the Spark Core<\/strong>. It allows developers to import relational data from Hive<\/a><\/strong> tables and parquet files, run SQL queries over imported data and existing RDDs and easily write RDDs out to Hive tables or Parquet files. As Spark SQL provides DataFrame APIs which performs the relational operation on both external data sources and Sparks built-in distributed collections. Spark SQL introduces extensible optimizer called Catalyst<\/strong> as it helps in supporting a wide range of data sources and algorithms in Bigdata<\/a><\/strong>.<\/p>\n

3. Apache Spark SQL Interfaces<\/h2>\n

Let’s discuss the interfaces of Apache Spark SQL in detail –<\/p>\n

i. DataFrame<\/h3>\n

Spark DataFrames evaluates lazily<\/a> like RDD Transformations<\/a><\/strong> in Apache Spark. A DataFrame is equivalent to the relational table in Spark SQL. A DataFrame stores the data into tables. DataFrame is similar\/identical to a table in a relational database but with richer optimization. It is a data abstraction and domain-specific language (DSL)<\/strong> applicable on the structure and semi-structured data. It is a distributed collection of data in the form of named column and row. For accessing data frames either SQL Context<\/strong> or Hive Context<\/strong> is needed. Learn about Spark SQL DataFrame in Detail<\/a>.<\/p>\n

ii. SQLContext<\/h3>\n

It’s the entry point for working with\u00a0structured data (rows and columns) in Apache Spark. It Allows the creation of DataFrame objects as well as the execution of SQL queries.<\/p>\n

iii. Hive Context<\/h3>\n

Working with Hive tables, a descendant of SQLContext. Hive Context is more battle-tested and provides a richer functionality than SQLContext.<\/p>\n

iv. DataSets<\/h3>\n

A Spark DataSet<\/strong> provides the benefits of RDDs<\/a>. For example, strongly-typed, immutable collection of objects that map to the relational schema already present (i.e. you can use a field of a row by name naturally row.columnName<\/em>). Datasets extend the benefit of compile-time type safety (It can analyze the applications for errors before they run). It also allows direct operations over user-defined classes. Learn about Apache Spark DataSet in detail<\/a>.<\/p>\n

v. JDBC Datasource<\/h3>\n

In Apache Spark, JDBC data source can read data from relational databases using JDBC API<\/strong>. It has preference over the RDD<\/em> because the data source returns the results as a DataFrame, can be handled in Spark SQL or joined beside other data sources.<\/p>\n

vi. Catalyst Optimizer<\/h3>\n

It is a functional programming construct in Scala<\/a><\/strong>. It is the newest and most technical component of Spark SQL. A catalyst<\/strong> is a query plan optimizer. It provides a general framework for transforming trees<\/em>, which performs analysis\/evaluation, optimization, planning, and runtime code spawning. Catalyst supports cost based optimization<\/strong> and rule-based optimization<\/strong>. It makes queries run much faster than their RDD counterparts. A catalyst is a rule-based modular library. Each rule in framework focuses on the distinct optimization. Learn Spark SQL Catalyst Optimizer in detail.<\/a><\/p>\n

4. Features of Apache Spark SQL<\/h2>\n

There are several features of Spark SQL which enhance which makes it a key component of Apache Spark framework.<\/p>\n

\"Features<\/a>

Features of Spark SQL<\/p><\/div>\n

i. Integrated<\/h3>\n

Logically mix SQL queries with Spark programs. Apache Spark SQL allows query structured data inside Spark programs, using SQL or a DataFrame API in Java, Scala, Python, and R<\/a><\/strong>.<\/em><\/p>\n

ii. Uniform Data Access<\/h3>\n

In Spark DataFrames and SQL supports a common way to access a variety of data sources, like Hive, Avro, Parquet, ORC, JSON, and JDBC.<\/em> Hence Spark SQL can join data across these sources.<\/p>\n

iii. Hive Compatibility<\/h3>\n

Runs unmodified Hive queries on current data. Spark SQL rewrites the Hive frontend and meta store, allowing full compatibility with current Hive data, queries, and UDFs.<\/p>\n

v. Standard Connectivity<\/h3>\n

Connect through JDBC or ODBC. A server that supports industry norms JDBC and ODBC connectivity for business intelligence tools.<\/p>\n

v. Performance & Scalability<\/h3>\n

Apache Spark SQL incorporates a cost-based optimizer, code generation, and columnar storage to make queries agile alongside computing thousands of nodes using the Spark engine, which provides full mid-query fault tolerance.<\/p>\n

5. Conclusion<\/h2>\n

Hence, Spark SQL allows Apache Spark to work with structured and unstructured data. The data can be from various sources. It enhances the performance of Spark applications. As a result, It provides an efficient platform to the Spark developers to work with various type of data.
\nIf in case you have any confusion about Apache Spark SQL, so leave a comment in a section below. We will be glad to solve them.
\nSee Also-<\/strong><\/p>\n