{"id":14667,"date":"2018-05-04T10:49:06","date_gmt":"2018-05-04T10:49:06","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=14667"},"modified":"2018-05-04T10:49:06","modified_gmt":"2018-05-04T10:49:06","slug":"kafka-workflow","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/","title":{"rendered":"Apache Kafka Workflow | Kafka Pub-Sub Messaging"},"content":{"rendered":"

In our last Kafka tutorial<\/strong>, we discussed Kafka Docker<\/strong>. Today, we will discuss Kafka Workflow.\u00a0 Also, we will cover Workflow of Pub-Sub Messaging along with Workflow of Queue Messaging \/ Consumer Group in detail. Also, we will see the role of ZooKeeper in Apache Kafka.<\/span><\/p>\n

So, let’s start with Kafka Workflow.<\/p>\n

What is Kafka Workflow?<\/span><\/h2>\n

In Kafka Workflow, Kafka is the collection of topics which are separated into one or more partitions and partition is a sequence of messages, where index identifies each message (also we call an offset). However, in a Kafka cluster<\/strong>, all the data is the disjoint union of partitions.<\/span><\/p>\n

The incoming messages are present at the end of a partition, hence consumer can read them. Also, by replicating the messages to different brokers, it maintains durability.<\/span><\/p>\n

In a very fast, reliable, persisted, fault-tolerance and zero downtime manner, Kafka offers a Pub-sub and queue-based messaging system. Moreover, producers send the message to a topic and the consumer can select any one of the message systems according to their wish.<\/span><\/p>\n

Workflow of Pub-Sub Messaging<\/span><\/h2>\n

In Apache Kafka, the stepwise workflow of the Pub-Sub Messaging is:<\/span><\/p>\n

    \n
  • At regular intervals, Kafka Producers<\/strong> send the message to a topic.<\/span><\/li>\n
  • Kafka Brokers<\/strong> stores all messages in the partitions configured for that particular topic, ensuring equal distribution of messages between partitions. For example, Kafka will store one message in the first partition and the second message in the second partition if the producer sends two messages and there are two partitions.<\/span><\/li>\n
  • Moreover, Kafka Consumer<\/strong> subscribes to a specific topic.<\/span><\/li>\n
  • Once the consumer subscribes to a topic, Kafka offers the current offset of the topic to the consumer and save the offset in the Zookeeper ensemble.<\/span><\/li>\n
  • Also, the consumer will request the Kafka in a regular interval, for new messages (like 100 Ms).<\/span><\/li>\n
  • Kafka will forward the messages to the consumers as soon as received from producers.<\/span><\/li>\n
  • The consumer will receive the message and process it.<\/span><\/li>\n
  • Then\u00a0Kafka broker receives an acknowledgment of the message processed.<\/span><\/li>\n
  • Further, the offset is changed and updated\u00a0to the new value\u00a0as soon as Kafka receives an acknowledgment. Even during server outrages, the consumer can read the next message correctly, because ZooKeeper maintains the offsets.<\/span><\/li>\n
  • However, until the consumer stops the request, the flow repeats.<\/span><\/li>\n
  • As a benefit, the consumer can rewind\/skip any offset of a topic at any time and also can read all the subsequent messages, as a par desire.<\/span><\/li>\n<\/ul>\n

    Workflow of Kafka Queue Messaging\/Consumer Group<\/span><\/h2>\n

    A group of Kafka consumers having the same Group ID can subscribe to a topic, instead of a single consumer, in a queue messaging system. <\/span><\/p>\n

    However, with the same Group ID all consumers, those are subscribing to a topic are considered as a single group and share the messages. This system\u2019s workflow is:<\/span><\/p>\n

      \n
    • In regular intervals, Kafka Producers send the message to a Kafka topic.<\/span><\/li>\n
    • As similar to the earlier scenario, here also\u00a0Kafka stores all messages in the partitions configured for that particular topic.<\/span><\/li>\n
    • Moreover, a single consumer in Kafka subscribes to a specific topic.<\/span><\/li>\n
    • In the same way as Pub-Sub Messaging, Kafka interacts with the consumer until new consumer subscribes to the same topic.<\/span><\/li>\n
    • As the new customers arrive, share mode starts in the operations and shares the data between two Kafka consumers. Moreover, until the number of Kafka consumers equals the number of partitions configured for that particular topic, the sharing repeats.<\/span><\/li>\n
    • Although, the new consumer in Kafka will not receive any further message, once the number of Kafka consumers exceeds the number of partitions. It happens until any one of the existing consumer unsubscribes. This scenario arises because in Kafka there is a condition that each Kafka consumer will\u00a0have a minimum of one partition and\u00a0if no partition remains blank, then new consumers will have to wait.<\/span><\/li>\n
    • In addition, we also call it Kafka Consumer Group. Hence, Apache Kafka will offer the best of both the systems in a very simple and efficient manner.<\/span><\/li>\n<\/ul>\n

      Role of ZooKeeper in Apache Kafka<\/span><\/h2>\n

      Apache Zookeeper serves as the coordination interface between the Kafka brokers and consumers. Also, we can say it is a distributed configuration and synchronization service. <\/span><\/p>\n

      Basically, ZooKeeper cluster shares the information with the Kafka servers. Moreover, Kafka stores basic metadata information in ZooKeeper Kafka, such as topics, brokers, consumer offsets (queue readers) and so on.<\/span><\/p>\n

      In addition, failure of Kafka Zookeeper\/broker does not affect the Kafka cluster. It is because the critical information which is stored in the ZooKeeper is replicated across its ensembles. Then\u00a0Kafka restores the state as ZooKeeper restarts, leading to zero downtime for Kafka. <\/span><\/p>\n

      However, Zookeeper also performs leader election between the Kafka brokers, in the cases of leadership failure.<\/span><\/p>\n

      Hence, this was all about Apache Kafka Workflow. Hope you like our explanation.<\/p>\n

      Conclusion<\/span><\/h2>\n

      Hence, we have seen the concept of Apache Kafka Workflow. Moreover, in this Kafka Workflow tutorial, we have discussed Workflow of Pub-Sub Messaging system, as well as the workflow of Kafka Queue<\/strong> Messaging\u00a0system. <\/span><\/p>\n

      Finally, we saw the role of Zookeeper in Apache Kafka. Still, if any doubt occurs regarding Kafka Workflow, feel free to ask in the comment section.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

      In our last Kafka tutorial, we discussed Kafka Docker. Today, we will discuss Kafka Workflow.\u00a0 Also, we will cover Workflow of Pub-Sub Messaging along with Workflow of Queue Messaging \/ Consumer Group in detail....<\/p>\n","protected":false},"author":5,"featured_media":14725,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[854,2941,7876,7974,10237,11630,16232],"class_list":["post-14667","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","tag-apache-kafka-workflow","tag-consumer-group","tag-kafka-consumers","tag-kafka-workflow","tag-pub-sub-messaging","tag-role-of-zookeeper-in-apache-kafka","tag-workflow-in-kafka"],"yoast_head":"\nApache Kafka Workflow | Kafka Pub-Sub Messaging - DataFlair<\/title>\n<meta name=\"description\" content=\"Apache Kafka workflow, workflow of pub-Sub messaging System, Workflow of Kafka Queue messaging \/ Consumer group system, role of Zookeeper in Apache Kafka\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/kafka-workflow\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Kafka Workflow | Kafka Pub-Sub Messaging - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Apache Kafka workflow, workflow of pub-Sub messaging System, Workflow of Kafka Queue messaging \/ Consumer group system, role of Zookeeper in Apache Kafka\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/kafka-workflow\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-04T10:49:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Work-Flow-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Kafka Workflow | Kafka Pub-Sub Messaging - DataFlair","description":"Apache Kafka workflow, workflow of pub-Sub messaging System, Workflow of Kafka Queue messaging \/ Consumer group system, role of Zookeeper in Apache Kafka","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/","og_locale":"en_US","og_type":"article","og_title":"Apache Kafka Workflow | Kafka Pub-Sub Messaging - DataFlair","og_description":"Apache Kafka workflow, workflow of pub-Sub messaging System, Workflow of Kafka Queue messaging \/ Consumer group system, role of Zookeeper in Apache Kafka","og_url":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-05-04T10:49:06+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Work-Flow-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Apache Kafka Workflow | Kafka Pub-Sub Messaging","datePublished":"2018-05-04T10:49:06+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/"},"wordCount":862,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Work-Flow-01-1.jpg","keywords":["Apache Kafka Workflow","Consumer group","Kafka Consumers","Kafka workflow","Pub-Sub messaging","role of zookeeper in Apache Kafka","workflow in Kafka"],"articleSection":["Apache Kafka Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/kafka-workflow\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/","url":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/","name":"Apache Kafka Workflow | Kafka Pub-Sub Messaging - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Work-Flow-01-1.jpg","datePublished":"2018-05-04T10:49:06+00:00","description":"Apache Kafka workflow, workflow of pub-Sub messaging System, Workflow of Kafka Queue messaging \/ Consumer group system, role of Zookeeper in Apache Kafka","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/kafka-workflow\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Work-Flow-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Work-Flow-01-1.jpg","width":1200,"height":628,"caption":"Apache Kafka Workflow"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/kafka-workflow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Kafka Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/kafka\/"},{"@type":"ListItem","position":3,"name":"Apache Kafka Workflow | Kafka Pub-Sub Messaging"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14667","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=14667"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14667\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/14725"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=14667"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=14667"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=14667"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}