This topic contains 2 replies, has 1 voice, and was last updated by  dfbdteam5 8 months, 1 week ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #6477

    dfbdteam5
    Moderator

    Explain Apache Spark Lineage Graph.
    What is RDD Lineage in Spark?

    #6478

    dfbdteam5
    Moderator

    When we apply a different transformation on RDD it creates RDD Linage graph. It is a new RDD from already existing RDDs. It is the dependencies graph between the existing and the new RDD formed. the need of RDD lineage graph arrives when we want to compute new RDD or if we want to recover the lost data from the lost persisted RDD.

    For more information study DAG in Spark

    #6479

    dfbdteam5
    Moderator

    Adding few more points on lineage graph:
    You can check lineage between two RDDs using rdd0.toDebugString. This gives back you the lineage graph from current rdd to all the previous dependencies of RDDs. See below. Whenever you see “+-” symbol from the toDebugString output, it means there will be next stage from the next operation onwards. This is indicates to identify that how many stage are created.

    scala> val rdd0 = sc.parallelize(List(“Ashok Vengala”,”Ashok Vengala”,”DataFlair”))
    rdd0: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[10] at parallelize at <console>:31

    scala> val count = rdd0.flatMap(rec => rec.split(” “)).map(word => (word,1)).reduceByKey(_+_)
    count: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[13] at reduceByKey at <console>:33

    scala> count.toDebugString
    res24: String =
    (2) ShuffledRDD[13] at reduceByKey at <console>:33 []
    +-(2) MapPartitionsRDD[12] at map at <console>:33 []
    | MapPartitionsRDD[11] at flatMap at <console>:33 []
    | ParallelCollectionRDD[10] at parallelize at <console>:31 []

    From down to up (i.e, last three rows): These will be performed in stage-0. And the first row(ShuffledRDD): this will operation will be performed in stage-1.

    In toDebugString output, we are seeing something like ParallelCollectionRDD, MapPartitionsRDD and ShuffleRDD. These are all implementation of RDD abstract class.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.