So, let’s start Spark Dataset Tutorial.<\/p>\n
![\"https:\/\/data-flair.training\/blogs\/apache-spark-dataset-tutorial\/\"](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/Spark-Dataset-01.jpg\")
<\/a>https:\/\/data-flair.training\/blogs\/apache-spark-dataset-tutorial\/<\/p><\/div>\n
2. What is Spark Dataset?<\/h2>\n
Dataset<\/strong> is a data structure in SparkSQL<\/a><\/strong> which is strongly typed and is a map to a relational schema. It represents structured queries with encoders. It is an extension to data frame API. Spark Dataset provides both type safety and object-oriented programming interface. We encounter the release of the dataset in Spark 1.6.<\/p>\n The encoder<\/strong> is primary concept in serialization<\/em> and deserialization<\/em> (SerDes<\/strong>) framework in Spark SQL. Encoders translate between JVM objects and Spark\u2019s internal binary format. Spark has built-in encoders which are very advanced. They generate bytecode to interact with off-heap<\/strong> data. Thus, Datasets provides a more functional programming interface to work with structured data.<\/p>\n To overcome the limitations of RDD<\/a> and Dataframe, Dataset emerged. In DataFrame, there was no provision for compile-time type safety<\/strong>. Data cannot be altered without knowing its structure. In RDD there was no automatic optimization. So for optimization, we do it manually when needed.<\/p>\n After having the introduction to dataSet, let’s now discuss various features of Spark Dataset-<\/p>\n Dataset in Spark provides Optimized query using Catalyst Query Optimizer<\/a> <\/strong>and Tungsten<\/strong>. Catalyst Query Optimizer<\/strong> is an execution-agnostic framework. It represents and manipulates a data-flow graph. Data flow graph is a tree of expressions and relational operators. By optimizing the Spark job Tungsten improves the execution. Tungsten emphasizes the hardware architecture of the platform on which Apache Spark runs.<\/p>\n Using Dataset we can check syntax and analysis at compile time. It is not possible using Dataframe, RDDs or regular SQL queries.<\/p>\n Spark Datasets are both serializable and Queryable. Thus, we can save it to persistent storage.<\/p>\n We can convert the Type-safe dataset to an \u201cuntyped\u201d DataFrame. To do this task Datasetholder provide three methods for conversion from Seq[T]<\/em> or RDD[T]<\/em> types to Dataset[T]:<\/em><\/p>\n The implementation of the Dataset is much faster than the RDD implementation. Thus increases the performance of the system. For the same performance using the RDD, the user manually considers how to express computation that parallelizes optimally.<\/p>\n While caching, it creates a more optimal layout. Spark knows the structure of data in the dataset.<\/p>\n It provides a single interface for Java<\/strong> and Scala<\/strong>. This unification ensures we can use Scala interface, code examples from both languages. It also reduces the burden of libraries. As libraries have no longer to deal with two different type of inputs.<\/p>\n To create a Dataset we need:<\/p>\n a. SparkSession<\/strong><\/p>\n SparkSession<\/em> is the entry point to the SparkSQL. It is a very first object that we create while developing Spark SQL applications using fully typed Dataset data abstractions. Using SparkSession.<\/strong>Builder, we can create an instance of SparkSession. And can stop SparkSession using the stop method (spark.stop<\/em><\/strong>). \u00a0<\/strong><\/p>\n b. QueryExecution<\/strong><\/p>\n We represent structured query execution pipeline of the dataset using QueryExecution<\/em>. To access QueryExecution of a Dataset use QueryExecution attribute. By executing a logical plan in Spark Session we get QueryExecution. c. Encoder<\/strong><\/p>\n An encoder <\/em>provides conversion between tabular representation and JVM objects. With the help of the encoder, we serialize the object. Encoder serializes objects for processing or transmitting over the network encoders.<\/p>\n So, this was all in Spark Dataset Tutorial. Hope you like our explanation.<\/p>\n Hence, in conclusion to Dataset, we can say it is a strongly typed data structure in Apache Spark. Moreover, it represents structured queries. Also, it fuses together the functionality of RDD<\/a> and DataFrame. We can generate the optimized query using Dataset. So, Dataset lessens the memory consumption and provides a single API for both Java and Scala.<\/p>\n If you like this post and feel that I have missed any point, so, do let me know by leaving a comment.<\/p>\n See Also-<\/strong><\/p>\n 1. Objective – Apache Spark Dataset Today, in\u00a0this blog on Apache Spark dataset, you can read all about what is dataset in Spark. Why the Spark DataSet needed, what is the encoder and what...<\/p>\n","protected":false},"author":6,"featured_media":43108,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[896,2405,3542,4545,8209,13048,16614,13058,13173],"class_list":["post-2685","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-apache-spark","tag-catalyst-optimizer-in-spark","tag-dataset-features","tag-faster-computation","tag-less-memory-consumption","tag-spark-dataset","tag-spark-dataset-tutorial","tag-spark-encoders","tag-sparksession"],"yoast_head":"\n
\nAn encoder provides on-demand access to individual attributes without having to de-serialize an entire object. To make input-output time and space efficient, Spark SQL uses the SerDe framework. Since encoder knows the schema of record, it can achieve serialization and deserialization.
\nSpark Dataset is structured and lazy query expression that triggers the action<\/strong>. Internally dataset represents a logical plan. The logical plan<\/strong> tells the computational query that we need to produce the data. the logical plan is a base catalyst query plan for the logical operator to form a logical query plan. When we analyze this and resolve we can form a physical query plan.
\nDataset clubs the features of RDD<\/a><\/strong> and DataFrame<\/a>.<\/strong> It provides:<\/p>\n\n
3. Need of Dataset in Spark<\/h2>\n
4. Features of Dataset in Spark<\/h2>\n
a. Optimized Query<\/h3>\n
b. Analysis at compile time<\/h3>\n
c.\u00a0Persistent Storage<\/h3>\n
d. Inter-convertible<\/h3>\n
\n
e. Faster Computation<\/h3>\n
f. Less Memory Consumption<\/h3>\n
g. Single API for Java and Scala<\/h3>\n
5. Creating Dataset<\/h2>\n
\nexecutePlan(plan: LogicalPlan): QueryExecution<\/em>
\nexecutePlan<\/strong>\u00a0executes the input\u00a0LogicalPlan\u00a0to produce a\u00a0QueryExecution in the current\u00a0SparkSession.<\/p>\n6. Conclusion<\/h2>\n
\n