{"id":2125,"date":"2017-04-20T11:37:11","date_gmt":"2017-04-20T11:37:11","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2125"},"modified":"2018-11-21T11:24:53","modified_gmt":"2018-11-21T05:54:53","slug":"limitations-of-apache-spark","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/limitations-of-apache-spark\/","title":{"rendered":"Limitations of Apache Spark – Ways to Overcome Spark Drawbacks"},"content":{"rendered":"

1. Objective<\/h2>\n

Some of the drawbacks of Apache Spark<\/strong> are there is no support for real-time processing, Problem with small file, no dedicated File management system, Expensive and much more due to these limitations of Apache Spark, industries have started shifting to Apache Flink<\/strong>– 4G of Big Data. <\/strong><\/a>
\nIn this Apache Spark limitations tutorial, we will discuss these Apache Spark disadvantages and how to overcome these limitations of Apache Spark.<\/p>\n

\"Limitations<\/a>

Limitations of Apache Spark – Ways to Overcome Spark Drawbacks<\/p><\/div>\n

2. Limitations of Apache Spark<\/h2>\n

As we know Apache Spark <\/a><\/strong>is the next Gen Big data tool that is being widely used by industries but there are certain limitations of Apache Spark due to which industries have started shifting to Apache Flink<\/strong><\/a>– 4G of Big Data<\/strong>. Before we learn what are the disadvantages of Apache Spark, let us learn the\u00a0advantages of Apache Spark<\/a>.<\/p>\n

So let us now understand Apache Spark problems and when not to use Spark.<\/p>\n

a. No Support for Real-time Processing<\/h3>\n

In Spark Streaming<\/strong><\/a>, the arriving live stream of data is divided into batches of the pre-defined interval, and each batch of data is treated like Spark Resilient Distributed Database (RDDs). <\/a><\/strong>Then these RDDs are processed using the operations like map, reduce, join etc. The result of these operations is returned in batches. \u00a0Thus, it is not real time processing but Spark is near real-time processing of live data. Micro-batch processing takes place in Spark Streaming.<\/p>\n

b. Problem with Small File<\/h3>\n

If we use Spark with Hadoop<\/a><\/strong>, we come across a problem of a small file. HDFS<\/strong><\/a> provides a limited number of large files rather than a large number of small files. Another place where Spark legs behind is we store the data gzipped in S3<\/strong>. This pattern is very nice except when there are lots of small gzipped files. Now the work of the Spark is to keep those files on network and uncompress them. The gzipped files can be uncompressed only if the entire file is on one core. So a large span of time will be spent in burning their core unzipping files in sequence.<\/p>\n

In the resulting RDD<\/strong>, each file will become a partition; hence there will be a large amount of tiny partition within an RDD. Now if we want efficiency in our processing, the RDDs should be repartitioned into some manageable format. This requires extensive shuffling over the network.<\/p>\n

c. No File Management System<\/h3>\n

Apache Spark does not have its own file management system, thus it relies on some other platform like Hadoop<\/strong> or another cloud-based platform which is one of the Spark known issues.<\/p>\n

d. Expensive<\/h3>\n

In-memory<\/strong><\/a> capability can become a bottleneck when we want cost-efficient processing of big data as keeping data in memory is quite expensive, the memory consumption is very high, and it is not handled in a user-friendly manner. Apache Spark requires lots of RAM to run in-memory, thus the cost of Spark is quite high.<\/p>\n

e. Less number of Algorithms<\/h3>\n

Spark MLlib<\/strong> lags behind in terms of a number of available algorithms like Tanimoto distance.<\/p>\n

f. Manual Optimization<\/h3>\n

The Spark job requires to be manually optimized and is adequate to specific datasets. If we want to partition and cache in Spark<\/strong><\/a> to be correct, it should be controlled manually.<\/p>\n

g. Iterative Processing<\/h3>\n

In Spark, the data iterates in batches and each iteration is scheduled and executed separately.<\/p>\n

h. Latency<\/h3>\n

Apache Spark has higher latency as compared to Apache Flink<\/strong><\/a>.<\/p>\n

i. Window Criteria<\/h3>\n

Spark does not support record based window criteria. It only has time-based window criteria.<\/p>\n

j. Back Pressure Handling<\/h3>\n

Back pressure is build up of data at an input-output when the buffer is full and not able to receive the additional incoming data. No data is transferred until the buffer is empty. Apache Spark is not capable of handling pressure implicitly rather it is done manually.
\nThese are some of the major pros and cons of Apache Spark. We can overcome these limitations of Spark by using Apache Flink \u2013 4G of Big Data<\/a>.<\/p>\n

3. Conclusion<\/h2>\n

Although Spark has many drawbacks, it is still popular in the market for big data <\/strong><\/a>solution. But there are various technologies that are overtaking Spark. Like stream processing is much better using Flink then Spark as it is real time processing.\u00a0Learn feature wise differences between Apache Spark vs Apache Flink<\/a><\/strong> to understand which is better and how.
\nSee Also-<\/strong><\/p>\n