{"id":2889,"date":"2017-06-17T06:15:19","date_gmt":"2017-06-17T06:15:19","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2889"},"modified":"2018-11-16T14:48:16","modified_gmt":"2018-11-16T09:18:16","slug":"spark-streaming-checkpoint","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/spark-streaming-checkpoint\/","title":{"rendered":"Spark Streaming Checkpoint in Apache Spark"},"content":{"rendered":"

1. Objective<\/h2>\n

In this blog of Apache Spark<\/strong> Streaming Checkpoint, you will read all about Spark Checkpoint. First of all, we will discuss What is Checkpointing in Spark, then, How Checkpointing helps to achieve Fault Tolerance in Apache Spark. Types of Apache Spark Checkpoint i.e Metadata Checkpointing, Data Checkpointing, the comparison between Checkpointing vs Persist() in Spark are also covered in this Spark tutorial.<\/p>\n

\"Spark<\/a>

Spark Streaming Checkpoint in Apache Spark<\/p><\/div>\n

If you want to learn Apache Spark from basics then our previous post on Apache Spark Introduction<\/strong><\/a> will help you. It has a Spark video which helps you to learn Apache Spark simply way.<\/p>\n

2. Introduction to Spark Streaming Checkpoint<\/h2>\n

The need with Spark Streaming<\/strong><\/a> application is that it should be operational 24\/7. Thus, the system should also be fault tolerant. If any data is lost, the recovery should be speedy. Spark streaming accomplishes this using checkpointing.<\/strong>
\nSo, Checkpointing<\/em> is a process to truncate RDD lineage graph<\/a>. It saves the application state timely to reliable storage (HDFS<\/strong><\/a>). As the driver restarts the recovery takes place.
\nThere are two types of data that we checkpoint in Spark:<\/em><\/p>\n

    \n
  1. Metadata Checkpointing –<\/strong>\u00a0Metadata<\/em> means the data about data. It refers to saving the metadata to fault tolerant storage like HDFS. Metadata includes configurations, DStream<\/strong> operations, and incomplete batches. Configuration refers to the configuration used to create streaming DStream operations<\/strong><\/a> are operations which define the steaming application. Incomplete batches are batches which are in the queue but are not complete.<\/li>\n
  2. Data Checkpointing –<\/strong>: It refers to save the RDD<\/a> <\/strong>to reliable storage because its need arises in some of the stateful transformations<\/em>. It is in the case when the upcoming RDD depends on the RDDs of previous batches. Because of this, the dependency keeps on increasing with time. Thus, to avoid such increase in recovery time the intermediate RDDs are periodically checkpointed to some reliable storage. As a result, it cuts down the dependency chain.<\/li>\n<\/ol>\n

    To set the Spark checkpoint directory call: SparkContext.setCheckpointDir(directory: String)<\/em><\/p>\n

    3. Types of Checkpointing in Apache Spark<\/h2>\n

    There are two types of Apache Spark checkpointing:<\/p>\n

      \n
    1. Reliable Checkpointing –<\/strong> It refers to that checkpointing in which the actual RDD is saved in reliable distributed file system, e.g. HDFS. To set the checkpoint directory call:\u00a0SparkContext.setCheckpointDir(directory: String)<\/em>. When running on the cluster the directory must be an HDFS path since the driver tries to recover the checkpointed RDD from a local file. While the checkpoint files are actually on the executor’s machines.<\/li>\n
    2. Local Checkpointing:<\/strong> In this checkpointing, in Spark Streaming<\/em> or GraphX<\/em> we truncate the RDD lineage graph in Spark. In this, the RDD is persisted to local storage in the executor.<\/li>\n<\/ol>\n

      4. Difference between Spark Checkpointing and Persist<\/h2>\n

      There are various differences between Spark Checkpoint vs Persist. Let’s discuss it one by one-
      \nPersist<\/strong><\/p>\n