{"id":2979,"date":"2017-06-22T12:42:39","date_gmt":"2017-06-22T12:42:39","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2979"},"modified":"2018-09-18T12:59:16","modified_gmt":"2018-09-18T07:29:16","slug":"spark-sql-performance-tuning","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/spark-sql-performance-tuning\/","title":{"rendered":"Spark SQL Performance Tuning – Learn Spark SQL"},"content":{"rendered":"

1. Objective<\/h2>\n

The Spark SQL performance can be affected by some tuning consideration. To represent our data efficiently, it uses the knowledge of types very effectively. Spark SQL plays a great role in the optimization of queries. This blog also covers what is Spark SQL performance tuning and various factors to tune the Spark SQL performance in Apache Spark<\/strong><\/a>.
\nBefore reading this blog I would recommend you to read Spark Performance Tuning<\/a>.<\/strong>\u00a0It will increase your understanding of Spark and help further in this blog.<\/p>\n

\"Spark<\/a>

Spark SQL Performance Tuning – Learn Spark SQL<\/p><\/div>\n

2. What is Spark SQL Performance Tuning?<\/h2>\n

Spark SQL<\/strong><\/a> is the module of Spark for structured data processing. The high-level query language and additional type information makes Spark SQL more efficient. Spark SQL translates commands into codes that are processed by executors. Some tuning consideration can affect the Spark SQL performance. To represent our data efficiently, it also uses the knowledge of types very effectively. Spark SQL plays a great role in the optimization of queries.
\nThe Spark SQL makes use of in-memory<\/a><\/strong> columnar storage while caching data. The in-memory columnar is a feature that allows storing the data in a columnar format, rather than row format. The columnar storage allows itself extremely well to analytic queries found in business intelligence product. Using columnar storage, the data takes less space when cached and if the query depends only on the subsets of data, thus Spark SQL minimizes the data read.<\/p>\n

3. Options for Performance Tuning in Spark SQL<\/h2>\n

There are several different Spark SQL performance tuning options are available:
\ni. spark.sql.codegen<\/strong>
\nThe default value of spark.sql.codegen<\/em> is false<\/strong>. When the value of this is true, Spark SQL will compile each query to Java bytecode very quickly. Thus, improves the performance for large queries. But the issue with codegen is that it slows down with very short queries. This happens because it has to run a compiler for each query.
\nii. spark.sql.inMemorycolumnarStorage.compressed<\/strong>
\nThe default value of spark.sql.inMemorycolumnarStorage.compressed<\/em> is true<\/strong>. When the value is true we can compress the in-memory columnar storage automatically based on statistics of the data.
\niii. spark.sql.inMemoryColumnarStorage.batchSize<\/strong>
\nThe default value of spark.sql.inMemoryColumnarStorage.batchSize<\/em> is 10000<\/strong>. It is the batch size for columnar caching. The larger values can boost up memory utilization but causes an out-of-memory problem.
\niv. spark.sql.parquet.compression.codec<\/strong>
\nThe\u00a0spark.sql.parquet.compression.codec<\/em>\u00a0uses default snappy compression. Snappy is a library which for compression\/decompression. It mainly aims at very high speed and reasonable compression. In most compression, the resultant file is 20 to 100% bigger than other inputs although it is the order of magnitude faster. Other possible option includes uncompressed, gzip and lzo.
\nNote:<\/strong>
\nIn Spark SQL as more optimizations<\/strong><\/a> are performed automatically, it is possible that following options can get vanished in the further release:<\/p>\n