{"id":2609,"date":"2017-05-19T12:39:52","date_gmt":"2017-05-19T12:39:52","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2609"},"modified":"2018-11-16T13:58:11","modified_gmt":"2018-11-16T08:28:11","slug":"apache-spark-rdd-persistence-caching","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-persistence-caching\/","title":{"rendered":"RDD Persistence and Caching Mechanism in Apache Spark"},"content":{"rendered":"

1. Objective<\/h2>\n

This blog covers the detailed view of\u00a0Apache Spark<\/a><\/strong>\u00a0RDD Persistence and Caching. This tutorial gives the answers for – What is RDD persistence, Why do we need to call cache<\/em><\/strong> or persist<\/em><\/strong> on an RDD,\u00a0<\/em>What\u00a0is the Difference between\u00a0Cache() and Persist() method in Spark, What are the different storage levels in spark to store the persisted RDD, How to Unpersist RDD?\u00a0The need of persisting RDD and various advantages of persistence are also discussed in this Spark tutorial.<\/p>\n

\"RDD<\/a>

RDD Persistence and Caching Mechanism in Apache Spark<\/p><\/div>\n

2. What is RDD Persistence and Caching in Spark?<\/h2>\n

Spark\u00a0RDD<\/a> <\/strong>persistence is an optimization technique in which saves the result of RDD evaluation. Using this we save the intermediate result so that we can use it further if required. It reduces the computation overhead.<\/span>
\nWe can make persisted RDD through cache()<\/strong> and persist()<\/strong> methods. When we use the cache() method we can store all the RDD in-memory. We can persist the RDD in memory and use it efficiently across parallel operations. <\/span>
\nThe difference between cache() and persist() is that using cache() the default storage level is MEMORY_ONLY<\/strong> while using persist() we can use various storage levels (described below).\u00a0<\/span>It is a key tool for an interactive algorithm. Because, when we persist RDD each node stores any partition of it that it computes in memory and makes it reusable for future use. This process speeds up the further computation ten times.<\/span>
\nWhen the RDD is computed for the first time, it is kept in memory on the node. The cache memory of the Spark is fault tolerant<\/a> so whenever any partition of RDD is lost, it can be recovered by transformation Operation<\/a> that originally created it.<\/span><\/p>\n

3. Need of Persistence in Apache Spark<\/h2>\n

In Spark, we can use some RDD\u2019s multiple times. If honestly, we repeat the same process of RDD<\/strong> evaluation<\/strong> each time it required or brought into action. This task can be time and memory consuming, especially for iterative algorithms that look at data multiple times. To solve the problem of repeated computation the technique of persistence came into the picture.<\/span><\/p>\n

4. Benefits of RDD Persistence in Spark<\/h2>\n

There are some advantages of RDD caching and persistence mechanism in spark. It makes the whole system<\/span><\/p>\n