Explain the level of parallelism in spark streaming.

Viewing 1 reply thread
  • Author
    Posts
    • #5390
      DataFlair Team
      Moderator

      Explain the level of parallelism in spark streaming.

    • #5391
      DataFlair Team
      Moderator

      > In order to reduce the processing time, one need to increase the parallelism.
      > In Spark Streaming, there are three ways to increase the parallelism :
      (1) Increase the number of receivers : If there are too many records for single receiver (single machine) to read in and distribute so that is bottleneck. So we can increase the no. of receiver depends on scenario.
      (2) Re-partition the receive data : If one is not in a position to increase the no. of receivers in that case redistribute the data by re-partitioning.
      (3) Increase parallelism in aggregation :

      for complete guide on Spark Streaming you may refer to Apache Spark-Streaming guide

Viewing 1 reply thread
  • You must be logged in to reply to this topic.