{"id":2603,"date":"2017-05-16T12:34:25","date_gmt":"2017-05-16T12:34:25","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2603"},"modified":"2018-11-16T17:38:29","modified_gmt":"2018-11-16T12:08:29","slug":"apache-spark-sql-dataframe-tutorial","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-spark-sql-dataframe-tutorial\/","title":{"rendered":"Spark SQL DataFrame Tutorial – An Introduction to DataFrame"},"content":{"rendered":"

1. Objective<\/h2>\n

In this Spark SQL DataFrame<\/strong> tutorial, we will learn what is DataFrame in Apache Spark<\/a><\/strong> and the need of Spark Dataframe. The tutorial covers the limitation of Spark RDD and How DataFrame overcomes those limitations. How to create DataFrame in Spark, Various Features of DataFrame like Custom <\/strong>Memory Management,\u00a0Optimized Execution plan,\u00a0<\/strong>and its limitations are also covers in this Spark tutorial.<\/p>\n

So, let’s start Spark SQL DataFrame tutorial.<\/p>\n

\"Spark<\/a>

Spark SQL DataFrame Tutorial – An Introduction to DataFrame<\/p><\/div>\n

2. What is Spark SQL DataFrame?<\/h2>\n

DataFrame<\/strong>\u00a0appeared in Spark Release 1.3.0. We can term DataFrame as Dataset organized into named columns. DataFrames are similar to the table in a relational database or data frame in R<\/a> \/Python. It can be said as a relational table with good optimization technique.
\nThe idea behind DataFrame is it allows processing of a large amount of structured data. DataFrame contains rows with Schema. The schema<\/strong> is the illustration of the structure of data.
\nDataFrame in Apache Spark prevails over RDD<\/a> but contains the features of RDD<\/a> as well. The features common to RDD and DataFrame are\u00a0immutability<\/strong>, in-memory<\/a><\/strong>, resilient, distributed computing capability. It allows the user to impose the structure onto a distributed collection of data. Thus provides higher level abstraction.
\nWe can build DataFrame from different data sources. For Example structured data file, tables in Hive<\/a>, external databases or existing RDDs. The Application Programming Interface (APIs) of DataFrame is available in various languages. Examples include\u00a0Scala<\/a>, Java, Python, and R.
\nBoth in Scala and Java, we represent DataFrame as Dataset of rows. In the Scala API, DataFrames are type alias of Dataset[Row]. In Java API, the user uses Dataset<Row> to represent a DataFrame.<\/p>\n

3. Why DataFrame?<\/h2>\n

DataFrame is one step ahead of RDD. <\/strong>Since\u00a0it provides memory management and optimized execution plan.
\na. Custom Memory Management:<\/strong> This is also known as Project Tungsten.<\/strong>\u00a0A lot of memory is saved as the data is stored in off-heap memory in binary format. Apart from this, there is no Garbage Collection overhead. Expensive Java serialization is also avoided. Since the data is stored in binary format and the schema of memory is known.
\nb. Optimized Execution plan:<\/strong> This is also known as the query optimizer<\/strong>. Using this, an optimized execution plan is created for the execution of a query. Once the optimized plan is created final execution takes place on RDDs of Spark.<\/p>\n

4. Features of Apache Spark DataFrame<\/h2>\n

Some of the limitations of Spark RDD<\/a> were-<\/p>\n