What is RDD lineage graph or linage operation in Apache Spark

[DataFlair Team]

Explain lineage graph operator in Apache Spark, how it enables fault-tolerance in Spark ?

[DataFlair Team]

The RDD Lineage Graph or RDD operator graph could be a graph of the entire parent RDDs of an RDD. It’s engineered as a result of materializing transformations to the RDD and then creating a logical execution set up.

The RDDs in Apache Spark rely on one or a lot of alternative RDDs. The illustration of dependencies in between RDDs is understood because of the lineage graph. Lineage graph info is employed to cipher every RDD on demand, so whenever a district of persistent RDD is lost, {the data | the info | the info} that’s lost will be recovered using the lineage graph information.

For details on RDD DAG refer to Directed Acyclic Graph

[DataFlair Team]

<li style=”list-style-type: none”>

Lineage Graph is the graph of all the parent RDDs for an

RDD

<li style=”list-style-type: none”>

.
• By applying a different transformation on an RDDresults in lineage graph.
• When one derives the new RDD from existing (previous) RDD using transformation, Spark keeps the track of all the dependencies between RDD is called lineage graph.
• Lineage Graph is useful for scenarios mentioned below:

(1) When there is a demand for computing the new RDD.
(2) To recover the lost data if part of persisted RDD is lost.

<li style=”list-style-type: none”>

• In other words, Lineage Graph is a graph of all transformation operation that needs to execute when an action operation is called.

For complete information on DAG cite on Directed Acyclic Graph in Apache Spark

[Author]

Posts