{"id":1924,"date":"2017-04-08T17:15:06","date_gmt":"2017-04-08T17:15:06","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=1924"},"modified":"2018-11-21T11:41:31","modified_gmt":"2018-11-21T06:11:31","slug":"dag-in-apache-spark","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/dag-in-apache-spark\/","title":{"rendered":"Directed Acyclic Graph DAG in Apache Spark"},"content":{"rendered":"

1. Objective<\/h2>\n

In this Apache Spark tutorial, we will understand what is DAG in Apache Spark, what is DAG Scheduler, what is the need of directed acyclic graph in Spark, how to create DAG in Spark and how it helps in achieving fault tolerance. We will also learn how DAG works in RDD, the advantages of DAG in Spark which creates the difference between Apache Spark and Hadoop MapReduce.<\/a><\/p>\n

(Directed Acyclic Graph) DAG<\/strong> in Apache Spark<\/strong> <\/a>is a set of Vertices<\/strong> and Edges<\/strong>, where vertices<\/em> represent the RDDs<\/strong> and the edges<\/em> represent the Operation to be applied on RDD<\/strong>. In Spark DAG, every edge directs from earlier to later in the sequence. On the calling of Action<\/em>, the created DAG submits to DAG Scheduler<\/strong> which further splits the graph into the stages <\/strong>of the task<\/strong>.<\/p>\n

\"Directed<\/a>

Directed Acyclic Graph DAG in Apache Spark<\/p><\/div>\n

2. What is DAG in Apache Spark?<\/h2>\n

DAG<\/strong>\u00a0a finite direct graph with no directed cycles. There are finitely many vertices<\/em> and edges,<\/em> where each edge directed from one vertex to another. It contains a sequence of vertices such that every edge is directed from earlier to later in the sequence. It is a strict generalization of MapReduce<\/strong><\/a> model. DAG operations can do better global optimization than other systems like MapReduce. The picture of DAG becomes clear in more complex jobs.<\/p>\n

Apache Spark DAG allows the user to dive into the stage and expand on detail on any stage. In the stage view,<\/em> the details of all RDDs<\/strong> <\/a>belonging to that stage are expanded. The Scheduler splits the Spark RDD into stages<\/strong> based on various transformation applied. (You can refer this link to learn RDD <\/a><\/p>\n

Transformations and Actions in detail<\/a>) Each stage is comprised of tasks<\/strong>, based on the partitions of the RDD, which will perform same computation in parallel. The graph here refers to navigation, and directed and acyclic refers to how it is done.<\/em><\/p>\n

3. Need of Directed Acyclic Graph in Spark<\/h2>\n

The limitations of Hadoop<\/a> MapReduce became a key point to introduce DAG in Spark. The computation through MapReduce in three steps:<\/p>\n