{"id":14821,"date":"2018-05-01T10:31:40","date_gmt":"2018-05-01T10:31:40","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=14821"},"modified":"2018-05-01T10:31:40","modified_gmt":"2018-05-01T10:31:40","slug":"kafka-spark-streaming-integration","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/","title":{"rendered":"Apache Kafka + Spark Streaming Integration"},"content":{"rendered":"

In order to build real-time applications, Apache Kafka\u00a0<\/strong>–\u00a0Spark\u00a0Streaming Integration are the best combinations. So, in this article, we will learn the whole concept of Spark Streaming Integration in Kafka in detail. Moreover, we will look at Spark <\/strong>Streaming-Kafka example. <\/span><\/p>\n

After this, we will discuss a receiver-based approach and a direct approach to\u00a0Kafka Spark Streaming Integration. Also, we will look advantages of direct approach to receiver-based approach in Kafka Spark Streaming Integration.<\/span><\/p>\n

So, let’s start Kafka Spark Streaming Integration<\/span><\/p>\n

What is Kafka Spark Streaming Integration?<\/span><\/h2>\n

In Apache Kafka Spark Streaming Integration, there are two approaches to configure Spark Streaming to receive data from Kafka i.e. Kafka Spark Streaming Integration.<\/p>\n

First is by using Receivers and Kafka\u2019s high-level API, and a second, as well as a new approach, is without using Receivers. There are different programming models for both the approaches, such as performance characteristics and semantics guarantees.<\/p>\n

\"Kafka-<\/a>

What is Kafka-Spark Streaming Integration<\/p><\/div>\n

Let\u2019s study both approaches in detail.<\/strong><\/p>\n

a. Receiver-Based Approach<\/span><\/h3>\n

Here, we use a Receiver to receive the data. So, by using the Kafka high-level consumer<\/strong> API, we implement the Receiver. Further, the received data is stored in Spark executors<\/strong>. Then jobs launched by Kafka – Spark Streaming processes the data.<\/span><\/p>\n

Although, it is a possibility that this approach can lose data under failures under default configuration. Hence, we have to additionally enable write-ahead logs in Kafka Spark Streaming, to ensure zero-data-loss. <\/span><\/p>\n

That saves all the received Kafka data into write-ahead logs on a distributed file system synchronously. In this way, it is possible to recover all the data on failure.<\/span><\/p>\n

Further, we will discuss how to use this Receiver-Based Approach in our Kafka Spark Streaming application.<\/span><\/p>\n

i.<\/strong>\u00a0<\/span>Linking<\/h4>\n

Now, link your Kafka streaming application with the following artifact, for Scala<\/strong>\/Java <\/strong>applications using SBT\/Maven project definitions.<\/span><\/p>\n

groupId = org.apache.spark\nartifactId = spark-streaming-kafka-0-8_2.11\nversion = 2.2.0<\/pre>\n
However, we will have to add this above library and its dependencies when deploying our application, for Python applications.<\/strong><\/span><\/p>\n
ii.<\/strong>\u00a0<\/span>Programming<\/h4>\n
Afterward, create an input DStream by importing KafkaUtils, in the streaming application code:<\/p>\n
import org.apache.spark.streaming.kafka._\nval kafkaStream = KafkaUtils.createStream(streamingContext,\n    [ZK quorum], [consumer group id], [per-topic number of Kafka partitions to consume])<\/pre>\n
Also, using variations of createStream, we can specify the key and value classes and their corresponding decoder classes. <\/span><\/p>\n
iii.<\/strong>\u00a0<\/span>Deploying<\/h4>\n
As with any Spark applications, spark-submit is used to launch your application. However, the details are slightly different for Scala\/Java applications and Python applications.<\/span><\/p>\n
Moreover, using –packages spark-streaming-Kafka-0-8_2.11 and its dependencies can be directly added to spark-submit, for Python applications, which lack SBT\/Maven project management.<\/span><\/p>\n
.\/bin\/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 ...\n<\/pre>\n
Also, we can also download the JAR of the Maven artifact spark-streaming-Kafka-0-8-assembly from the Maven repository. Then add it to spark-submit with –jars.<\/p>\n
b. Direct Approach (No Receivers)<\/span><\/h3>\n
After Receiver-Based Approach, new receiver-less \u201cdirect\u201d approach has been introduced. It ensures stronger end-to-end guarantees. This approach periodically queries Kafka for the latest offsets in each topic+partition, rather than using receivers to receive data. <\/span><\/p>\n
Also, defines the offset ranges to process in each batch, accordingly. Moreover, to read the defined ranges of offsets from Kafka, it\u2019s simple consumer API is used, especially when the jobs to process the data are launched. However, it is similar to read files from a file system.<\/span>
\n<\/b><\/p>\n
Note:<\/b> This feature was introduced in Spark 1.3 for the Scala and Java API, in Spark 1.4 for the Python API.<\/span><\/p>\n
Now, let\u2019s discuss how to use this approach in our streaming application.<\/span><\/p>\n
To learn more about Consumer API follow the below link:<\/p>\n
i.\u00a0Linking<\/strong><\/h4>\n
However, this approach is supported only in Scala\/Java<\/strong> application. With the following artifact, link the SBT\/Maven project.<\/span><\/p>\n
groupId = org.apache.spark\nartifactId = spark-streaming-kafka-0-8_2.11\nversion = 2.2.0<\/pre>\n
ii.\u00a0Programming<\/strong><\/h4>\n
Further, import KafkaUtils and create an input DStream, in the streaming application code:<\/span><\/p>\n
import org.apache.spark.streaming.kafka._\nval directKafkaStream = KafkaUtils.createDirectStream[\n    [key class], [value class], [key decoder class], [value decoder class] ](\n    streamingContext, [map of Kafka parameters], [set of topics to consume])\n<\/pre>\n
We must specify either metadata.broker.list or bootstrap.servers, in the Kafka parameters. Hence, it will start consuming from the latest offset of each Kafka partition, by default. Although, it will start consuming from the smallest offset if you set configuration auto.offset.reset in Kafka parameters to smallest.<\/p>\n
Moreover, using other variations of KafkaUtils.createDirectStream we can start consuming from an arbitrary offset. Afterward, do the following to access the Kafka offsets consumed in each batch.<\/span><\/p>\n
\/\/ Hold a reference to the current offset ranges, so downstream can use it\nvar offsetRanges = Array.empty[OffsetRange]\ndirectKafkaStream.transform { rdd =>\n  offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges\n  rdd\n}.map {\n          ...\n}.foreachRDD { rdd =>\n  for (o <- offsetRanges) {\n    println(s\"${o.topic} ${o.partition} ${o.fromOffset} ${o.untilOffset}\")\n  }\n  ...\n}<\/pre>\n
If we want Zookeeper-based Kafka monitoring tools<\/strong> to show the progress of the streaming application, we can use this to update Zookeeper ourself.<\/span><\/p>\n
iii.\u00a0Deploying<\/strong><\/h4>\n
Here, deploying process is similar to deploying process of Receiver-Based Approach. <\/span><\/p>\n
Advantages of Direct Approach<\/span><\/h2>\n
There are\u00a0following advantages of 2nd approach over 1st approach in Spark Streaming Integration with Kafka:<\/span><\/p>\n
\"Kafka-<\/a>
Advantages of Direct Approach in Spark Streaming Integration with Kafka<\/p><\/div>\n
a. Simplified Parallelism<\/span><\/h3>\n
There is no requirement to create multiple input Kafka streams <\/strong>and union them. However, Kafka – Spark Streaming will create as many RDD partitions<\/strong> as there are Kafka partitions to consume, with the direct stream.<\/span><\/p>\n
 That will read data from Kafka in parallel. Hence, we can say, it is a one-to-one mapping between Kafka and RDD partitions, which is easier to understand and tune.<\/span><\/p>\n
b. Efficiency <\/span><\/h3>\n
Achieving zero-data-loss in the first approach required the data to be stored in a write-ahead\u00a0log, which further replicated the data. This is actually inefficient as the data effectively gets replicated twice – once by Kafka, and a second time by the write-ahead\u00a0log. <\/span><\/p>\n
The second approach eliminates the problem as there is no receiver, and hence no need for write-ahead\u00a0logs. As long as\u00a0we have sufficient Kafka retention, it is possible to recover messages from Kafka.<\/span><\/p>\n
c. Exactly-Once Semantics<\/span><\/h3>\n
Basically, we used Kafka\u2019s high-level API to store consumed offsets in Zookeeper in the first approach. However, to consume data from Kafka this is a traditional way. Even if it can ensure zero data loss, there is a small chance some records may get consumed twice under some failures.<\/span><\/p>\n
 It happens due to inconsistencies between data reliably received by Kafka – Spark Streaming and offsets tracked by Zookeeper<\/strong>. Therefore, we use a simple Kafka API that does not use Zookeeper, in this second approach.<\/span><\/p>\n
Thus each record is received by Spark Streaming effectively exactly once despite failures. <\/span><\/p>\n
Hence, make sure our output operation that saves the data to an external data store must be either idempotent or an atomic transaction that saves results and offsets. That helps to achieve exactly-once semantics for the output of our results.<\/span><\/p>\n
Although, there is one disadvantage also, that it does not update offsets in Zookeeper, thus Zookeeper-based Kafka monitoring tools will not show progress. But still, we can access the offsets processed by this approach in each batch and update Zookeeper yourself.<\/span><\/p>\n
So, this was all about Apache Kafka Spark Streaming Integration. Hope you like our explanation.<\/p>\n
Conclusion<\/span><\/h2>\n
Hence, in this Kafka- Spark Streaming Integration, we have learned the whole concept of Spark Streaming Integration with Apache Kafka in detail. Also, we discussed two different approaches for Kafka Spark Streaming configuration and that are Receiving Approach and Direct Approach.<\/p>\n
Moreover, we discussed the advantages of the Direct Approach. Furthermore, if any doubt occurs, feel free to ask in the comment\u00a0section.<\/p>\n","protected":false},"excerpt":{"rendered":"
In order to build real-time applications, Apache Kafka\u00a0–\u00a0Spark\u00a0Streaming Integration are the best combinations. So, in this article, we will learn the whole concept of Spark Streaming Integration in Kafka in detail. Moreover, we will...<\/p>\n","protected":false},"author":5,"featured_media":73715,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[335,825,3928,7841,7842,7979,7980,7986,11421,13129,13130,13132,13890,15964],"class_list":["post-14821","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","tag-advantages-of-direct-approach","tag-apache-kafka","tag-direct-approach","tag-kafka","tag-kafka-spark-streaming","tag-kafka-spark-integration","tag-kafka-spark-streaming-configuration","tag-kafka-spark-streaming-tutorial","tag-receiver-based-approach","tag-spark-straming-kafka-example","tag-spark-streaming","tag-spark-streaming-kafka","tag-streaming-application","tag-what-is-spark-streaming"],"yoast_head":"\nApache Kafka + Spark Streaming Integration - DataFlair<\/title>\n<meta name=\"description\" content=\"Kafka+Spark Streaming, Kafka- Spark Streaming Integration, Receiving Approach, Direct Approach, advantages of direct approach ,Spark Streaming Kafka example\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Kafka + Spark Streaming Integration - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Kafka+Spark Streaming, Kafka- Spark Streaming Integration, Receiving Approach, Direct Approach, advantages of direct approach ,Spark Streaming Kafka example\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-01T10:31:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/kafka-spark-streaming-integration.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"802\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Kafka + Spark Streaming Integration - DataFlair","description":"Kafka+Spark Streaming, Kafka- Spark Streaming Integration, Receiving Approach, Direct Approach, advantages of direct approach ,Spark Streaming Kafka example","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/","og_locale":"en_US","og_type":"article","og_title":"Apache Kafka + Spark Streaming Integration - DataFlair","og_description":"Kafka+Spark Streaming, Kafka- Spark Streaming Integration, Receiving Approach, Direct Approach, advantages of direct approach ,Spark Streaming Kafka example","og_url":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-05-01T10:31:40+00:00","og_image":[{"width":802,"height":420,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/kafka-spark-streaming-integration.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Apache Kafka + Spark Streaming Integration","datePublished":"2018-05-01T10:31:40+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/"},"wordCount":1171,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/kafka-spark-streaming-integration.jpg","keywords":["advantages of direct approach","Apache Kafka","Direct Approach","kafka","Kafka - Spark Streaming","Kafka- Spark Integration","Kafka- Spark Streaming configuration","kafka-spark streaming tutorial","receiver based approach","spark straming kafka example","spark streaming","spark streaming kafka","streaming application","what is spark streaming"],"articleSection":["Apache Kafka Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/","url":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/","name":"Apache Kafka + Spark Streaming Integration - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/kafka-spark-streaming-integration.jpg","datePublished":"2018-05-01T10:31:40+00:00","description":"Kafka+Spark Streaming, Kafka- Spark Streaming Integration, Receiving Approach, Direct Approach, advantages of direct approach ,Spark Streaming Kafka example","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/kafka-spark-streaming-integration.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/kafka-spark-streaming-integration.jpg","width":802,"height":420,"caption":"kafka spark streaming integration"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/kafka-spark-streaming-integration\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Kafka Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/kafka\/"},{"@type":"ListItem","position":3,"name":"Apache Kafka + Spark Streaming Integration"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14821","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=14821"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14821\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/73715"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=14821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=14821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=14821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}