Explain fold() operation in Spark.

[DataFlair Team]

Explain fold() operation in Spark.

[DataFlair Team]

• fold() is an action. It is wide operation (i.e. shuffle data across multiple partitions and output a single value)
• It takes function as an input which has two parameters of the same type and outputs a single value of the input type.
• It is similar to reduce but has one more argument ‘ZERO VALUE’ (say initial value) which will be used in the initial call on each partition.

def fold(zeroValue: T)(op: (T, T) ⇒ T): T

Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral “zero value”. The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object allocation; however, it should not modify t2.

This behaves somewhat differently from fold operations implemented for non-distributed collections in functional languages like Scala. This fold operation may be applied to partitions individually, and then fold those results into the final result, rather than apply the fold to each element sequentially in some defined ordering. For functions that are not commutative, the result may differ from that of a fold applied to a non-distributed collection.

zeroValue: The initial value for the accumulated result of each partition for the op operator, and also the initial value for the combine results from different partitions for the op operator – this will typically be the neutral element (e.g. Nil for list concatenation or 0 for summation)
Op: an operator used to both accumulate results within a partition and combine results from different partitions

Example :

val rdd1 = sc.parallelize(List(1,2,3,4,5),3)
rdd1.fold(5)(_+_)

Output :
Int = 35

val rdd1 = sc.parallelize(List(1,2,3,4,5))
rdd1.fold(5)(_+_)

Output :
Int = 25

val rdd1 = sc.parallelize(List(1,2,3,4,5),3)
rdd1.fold(3)(_+_)

Int = 27

For more operations on Apache Spark RDD refer Transformation and Action in Spark

[Author]

Posts