What is lineage graph in Apache Spark?

Viewing 2 reply threads
  • Author
    Posts
    • #5047
      DataFlair TeamDataFlair Team
      Spectator

      Explain Apache Spark Lineage Graph.
      What is RDD Lineage in Spark?

    • #5048
      DataFlair TeamDataFlair Team
      Spectator

      When we apply a different transformation on RDD it creates RDD Linage graph. It is a new RDD from already existing RDDs. It is the dependencies graph between the existing and the new RDD formed. the need of RDD lineage graph arrives when we want to compute new RDD or if we want to recover the lost data from the lost persisted RDD.

      For more information study DAG in Spark

    • #5049
      DataFlair TeamDataFlair Team
      Spectator

      Adding few more points on lineage graph:
      You can check lineage between two RDDs using rdd0.toDebugString. This gives back you the lineage graph from current rdd to all the previous dependencies of RDDs. See below. Whenever you see “+-” symbol from the toDebugString output, it means there will be next stage from the next operation onwards. This is indicates to identify that how many stage are created.

      scala> val rdd0 = sc.parallelize(List(“Ashok Vengala”,”Ashok Vengala”,”DataFlair”))
      rdd0: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[10] at parallelize at <console>:31

      scala> val count = rdd0.flatMap(rec => rec.split(” “)).map(word => (word,1)).reduceByKey(_+_)
      count: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[13] at reduceByKey at <console>:33

      scala> count.toDebugString
      res24: String =
      (2) ShuffledRDD[13] at reduceByKey at <console>:33 []
      +-(2) MapPartitionsRDD[12] at map at <console>:33 []
      | MapPartitionsRDD[11] at flatMap at <console>:33 []
      | ParallelCollectionRDD[10] at parallelize at <console>:31 []

      From down to up (i.e, last three rows): These will be performed in stage-0. And the first row(ShuffledRDD): this will operation will be performed in stage-1.

      In toDebugString output, we are seeing something like ParallelCollectionRDD, MapPartitionsRDD and ShuffleRDD. These are all implementation of RDD abstract class.

Viewing 2 reply threads
  • You must be logged in to reply to this topic.