This topic contains 1 reply, has 1 voice, and was last updated by  dfbdteam5 10 months ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #5450

    dfbdteam5
    Moderator

    Explain pipe() operation in Apache Spark

    #5452

    dfbdteam5
    Moderator
      <li style=”list-style-type: none”>
    • It is a transformation.


    def pipe(command: String): RDD[String]
    Return an RDD created by piping elements to a forked external process.

    • In general, Spark is using Scala, Java, and Python to write the program. However, if that is not enough, and one want to pipe (inject) the data which written in other languages like ‘R’, Spark provides general mechanism in the form of pipe() method
    • Spark provides the pipe() method on RDDs.
    • With Spark’s pipe() method, one can write a transformation of an RDD that can read each element in the RDD from standard input as String.
    • It can write the results as String to the standard output.

    For more transformation on RDDs see: Apache Spark Operations

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.