{"id":16390,"date":"2018-06-04T04:00:51","date_gmt":"2018-06-04T04:00:51","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=16390"},"modified":"2018-06-04T04:00:51","modified_gmt":"2018-06-04T04:00:51","slug":"kafka-queuing","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/","title":{"rendered":"Kafka Queuing: Apache Kafka as a Messaging System"},"content":{"rendered":"

In this Apache Kafka tutorial, we will learn the concept of\u00a0Apache Kafka<\/strong>\u00a0Queuing. Basically, Queuing in Kafka is one of the models for messaging traditionally. <\/span><\/p>\n

So, let\u2019s begin with the brief introduction to Kafka as a Messaging System, that will help us to understand the Kafka Queuing well. Moreover, we will see some of the applications of Kafka Queue to clear the concept better.<\/span><\/p>\n

So, let’s start with Kafka Queuing tutorial.<\/p>\n

Kafka as a Messaging System<\/span><\/h2>\n

There are two models for messaging traditionally, such as Kafka queuing and publish-subscribe in Kafka.<\/span>
\nLet\u2019s discuss both Kafka models in detail:<\/span><\/p>\n

i. Kafka Queue<\/span><\/h3>\n

In this Kafka messaging system, a pool of\u00a0Kafka consumers<\/strong>\u00a0may read from a server. Also, each record goes to one of them here. It has some strengths as well as some weakness. <\/span><\/p>\n

Its strength is that it permits us to divide up the processing of data over multiple consumer instances, that help us scale our processing. But its weakness is, it is not multi-subscriber, as soon as one process reads the data it’s gone.<\/span><\/p>\n

ii. Kafka Publish-Subscribe<\/span><\/h3>\n

Whereas in this Kafka Publish-Subscribe system, the record is broadcast to all the Kafka consumers. It permits us to broadcast data to multiple processes. However, it also has some limitations, like there is no way of scaling processing because here every message goes to every subscriber.<\/span><\/p>\n

In Kafka, these two concepts are generalized by the Kafka\u00a0consumer group. However, the consumer group in Kafka permits us to divide up processing over a collection of processes, with a Kafka queue. Here the collection of processes refers to the members of the consumer group. <\/span><\/p>\n

Moreover, Kafka permits us to broadcast messages to multiple consumer groups, with Kafka publish-subscribe.<\/span><\/p>\n

So, the main benefit of Kafka’s model is both these properties are available in every Kafka topic \u2014it can scale processing as well as it is multi-subscriber. Hence, that implies we do not have to select one or the other.<\/span><\/p>\n

Kafka has stronger ordering guarantees than a traditional messaging system, too.<\/span><\/p>\n

Since there is no parallelism in the processing in the traditional system, Kafka performs it well with the notion of parallelism. Because Kafka can offer both ordering guarantees and load balancing over a pool of consumer processes.<\/span><\/p>\n

Need for Kafka Queuing<\/span><\/h2>\n

One of Kafka Queuing application is Microservice architecture. It essentially demands some sort of message queuing system. Let\u2019s understand microservice architecture first.<\/p>\n

It is a concept of decoupling an interconnected monolithic application into different independent modules as well as external data sources along with APIs.<\/p>\n

Thus, for the purpose of handling microservices-external-source and inter-microservice and communications, Message queuing comes into the picture.<\/p>\n

In addition, while we divide a big monolithic application into smaller, microservices (loosely-coupled), at that time the REST API calls increases which are amongst those microservices, and the number of connections to external data sources also increases.<\/span><\/p>\n

However, keeping this huge system synchronous is not desirable,\u00a0because it can render the entire application unresponsive. Also, it can defeat the whole purpose of\u00a0dividing into microservices in the first place.<\/span><\/p>\n

Hence, having Kafka at that time makes the whole data flow easier. Because\u00a0it is distributed, highly fault-tolerant and also it has constant monitoring of broker nodes through services like Zookeeper. So, it makes it efficient to work.<\/span><\/p>\n

Message Queuing in ML Solutions Pipeline<\/span><\/h2>\n

Apart from that, we can also use Kafka queuing for the various ML solution pipelines. However, ML solutions are built as:<\/span>
\nA user interface (on the client side, mobile\/web) \u2014 \u2192An API server and the database \u2014 \u2192Machine learning (blackbox).<\/strong><\/p>\n

However, ML black box\u00a0is very compute-heavy and also it\u2019s not practically easy to have those requests on a blocking synchronous mode. Moreover, all the requests can be\u00a0in a queue and configure\u00a0consumer API to take those requests one by one and feed them into the ML black box, in this scenario.<\/span><\/p>\n

So, while it comes to compute-intensive tasks, this pipeline can easily handle such as recognizing objects from thousands of images, that might take considerable time even without missing any requests.<\/span><\/p>\n

Basically, microservices deployed into containers, such as by fault-tolerant distributed clusters of\u00a0Kafka broker<\/strong>\u00a0nodes it is mediated and using a Zookeeper it monitors it, which seems like a new way of enterprising software development.<\/span><\/p>\n

So, this was all about Kafka Queuing: Kafka as a messaging system. Hope you like our explanation.<\/p>\n

Conclusion<\/span><\/h2>\n

Hence, we have seen the complete concept of Kafka Queuing. Moreover, we discussed why Kafka as a message queuing. Along with this, we learned the two models- Kafka queue and Kafka publish-subscribe.<\/p>\n

Also, we saw why we need Kafka Queuing. Lastly, we discussed message queuing in the ML solution pipeline. However, if any query occurs regarding Queuing in Kafka, feel free to ask through the comment section.<\/p>\n","protected":false},"excerpt":{"rendered":"

In this Apache Kafka tutorial, we will learn the concept of\u00a0Apache Kafka\u00a0Queuing. Basically, Queuing in Kafka is one of the models for messaging traditionally. So, let\u2019s begin with the brief introduction to Kafka as...<\/p>\n","protected":false},"author":5,"featured_media":16798,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[7849,7924,7925,7926,8703,9028,11120,16160],"class_list":["post-16390","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","tag-kafka-as-a-message-system","tag-kafka-publish-subscribe","tag-kafka-queue","tag-kafka-queuing","tag-microservice-architecture","tag-need-for-kafka-queuing","tag-queuing-in-kafka","tag-why-kafka-queuing"],"yoast_head":"\nKafka Queuing: Apache Kafka as a Messaging System - DataFlair<\/title>\n<meta name=\"description\" content=\"need for Kafka queuing,kafka as a message system,kafka queue, Kafka Publish-subscribe,Message Queuing in ML Solutions Pipeline,Microservice architecture\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/kafka-queuing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Queuing: Apache Kafka as a Messaging System - DataFlair\" \/>\n<meta property=\"og:description\" content=\"need for Kafka queuing,kafka as a message system,kafka queue, Kafka Publish-subscribe,Message Queuing in ML Solutions Pipeline,Microservice architecture\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/kafka-queuing\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-04T04:00:51+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Queuing-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kafka Queuing: Apache Kafka as a Messaging System - DataFlair","description":"need for Kafka queuing,kafka as a message system,kafka queue, Kafka Publish-subscribe,Message Queuing in ML Solutions Pipeline,Microservice architecture","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/","og_locale":"en_US","og_type":"article","og_title":"Kafka Queuing: Apache Kafka as a Messaging System - DataFlair","og_description":"need for Kafka queuing,kafka as a message system,kafka queue, Kafka Publish-subscribe,Message Queuing in ML Solutions Pipeline,Microservice architecture","og_url":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-06-04T04:00:51+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Queuing-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Kafka Queuing: Apache Kafka as a Messaging System","datePublished":"2018-06-04T04:00:51+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/"},"wordCount":810,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Queuing-01-1.jpg","keywords":["Kafka as a message system","kafka publish-subscribe","Kafka queue","Kafka Queuing","Microservice architecture","need for kafka queuing","queuing in kafka","why kafka queuing"],"articleSection":["Apache Kafka Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/kafka-queuing\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/","url":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/","name":"Kafka Queuing: Apache Kafka as a Messaging System - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Queuing-01-1.jpg","datePublished":"2018-06-04T04:00:51+00:00","description":"need for Kafka queuing,kafka as a message system,kafka queue, Kafka Publish-subscribe,Message Queuing in ML Solutions Pipeline,Microservice architecture","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/kafka-queuing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Queuing-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Queuing-01-1.jpg","width":1200,"height":628,"caption":"Kafka Queuing"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/kafka-queuing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Kafka Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/kafka\/"},{"@type":"ListItem","position":3,"name":"Kafka Queuing: Apache Kafka as a Messaging System"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/16390","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=16390"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/16390\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/16798"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=16390"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=16390"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=16390"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}