Explain the operation reduce()

[DataFlair Team]

Explain the operation reduce()

[DataFlair Team]

> reduce() is an action. It is wide operation (i.e. shuffle data across multiple partitions and output a single value)
> It takes function as an input which has two parameter of the same type and output a single value of the input type.
> i.e. combine the elements of RDD together.

Example 1 :
val rdd1 = sc.parallelize(1 to 100)
val rdd2 = rdd1.reduce((x,y) => x+y)

OR

val rdd2 = rdd1.reduce(_ + _)

Output :
rdd2: Int = 5050

Example 2:
val rdd1 = sc.parallelize(1 to 5)
val rdd2 = rdd1.reduce(_*_)

Output :
rdd2: Int = 120

[DataFlair Team]

From :
http://data-flair.training/blogs/rdd-transformations-actions-apis-apache-spark/#33_Reduce

It takes function with two arguments an accumulator and a value which should be commutative and Associative in mathematical nature. It reduces a list of element s into one as a result. This function produces same result when continuously applied on same set of RDD data with multiple partitions irrespective of elements order. It is wide operation.

[Author]

Posts