{"id":2891,"date":"2017-06-17T09:13:24","date_gmt":"2017-06-17T09:13:24","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2891"},"modified":"2018-11-16T14:47:09","modified_gmt":"2018-11-16T09:17:09","slug":"apache-spark-dstream-discretized-streams","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-spark-dstream-discretized-streams\/","title":{"rendered":"Apache Spark DStream (Discretized Streams)"},"content":{"rendered":"

1. Objective<\/h2>\n

This Spark tutorial, walk you through the Apache Spark<\/strong><\/a> DStream<\/strong>. First of all, we will see what is Spark Streaming, then, what is DStream in Apache Spark. Discretized Stream Operations i.e Stateless and Stateful Transformations, Output operation, Input DStream, and Receivers are also discussed in this Apache Spark blog.<\/p>\n

\"Apache<\/a>

Apache Spark DStream (Discretized Streams)<\/p><\/div>\n

2. Introduction to DStream in Apache Spark<\/h2>\n

In this section, we will learn about DStream. What are its role, and responsibility in Spark Streaming? It includes what all methods are inculcated to deal with live streaming of data.
\nAs an extension to Apache Spark API, Spark Streaming<\/strong><\/a> is fault tolerant, high throughput system. It processes the live stream of data. Spark Streaming takes input from various reliable inputs sources like Flume<\/a>, HDFS<\/a>,<\/strong> and Kafka<\/strong> etc. and then sends the processed data to filesystems, database or live dashboards. The input data stream is divided into the batches of data and then generates the final stream of the result in batches.<\/p>\n

Spark DStream (Discretized Stream)<\/strong> is the basic abstraction of Spark Streaming<\/strong>. DStream is a continuous stream of data. It receives input from various sources like Kafka, Flume, Kinesis, or TCP sockets.<\/em> It can also be a data stream generated by transforming the input stream. At its core, DStream is a continuous stream of RDD (Spark abstraction)<\/em><\/strong>. Every RDD in DStream contains data from the certain interval.<\/p>\n

Any operation on a DStream applies to all the underlying RDDs. DStream covers all the details. It provides the developer a high-level API for convenience. As a result, Spark DStream facilitates working with streaming data.<\/p>\n

Spark Streaming offers fault-tolerance <\/a>properties for DStreams as that for RDDs. as long as a copy of the input data is available, it can recompute any state from it using the lineage of the RDDs<\/a>. By default, Spark replicates data on two nodes. As a result, Spark Streaming can bear single worker failures.<\/p>\n

3. Apache Spark DStream Operations<\/h2>\n

Like RDD, Spark DStream also support two types of Operations: Transformations and output Operations-<\/p>\n

i. Transformation<\/h3>\n

There are two types of transformation in DStream:<\/p>\n