{"id":14633,"date":"2018-04-25T15:01:20","date_gmt":"2018-04-25T15:01:20","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=14633"},"modified":"2018-04-25T15:01:20","modified_gmt":"2018-04-25T15:01:20","slug":"kafka-architecture","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/","title":{"rendered":"Kafka Architecture and Its Fundamental Concepts"},"content":{"rendered":"

In our last Kafka Tutorial<\/strong>, we discussed Kafka Use Cases and Applications<\/strong>. Today, in this Kafka Tutorial, we will discuss Kafka Architecture. In this Kafka Architecture article, we will see API’s in Kafka. <\/span><\/p>\n

Moreover, we will learn about Kafka Broker, Kafka Consumer, Zookeeper, and Kafka Producer. Also, we will see some fundamental concepts of Kafka.<\/span><\/p>\n

So, let’s start Apache Kafka Architecture.<\/p>\n

Kafka Architecture – Apache Kafka APIs<\/span><\/h2>\n

Apache Kafka Architecture has four core APIs, producer API, Consumer API, Streams API, and Connector API. Let’s discuss them one by one:<\/p>\n

\"Kafka<\/a>

Kafka Architecture – Apache Kafka APIs<\/p><\/div>\n

a.\u00a0<\/span>Producer API<\/h3>\n

In order to publish a stream of records to one or more Kafka topics, the Producer API<\/span> allows an application.\u00a0<\/span><\/p>\n

b.\u00a0<\/span>Consumer API<\/h3>\n

This API permits an application to subscribe to one or more topics and also to process the stream of records produced to them. <\/span><\/p>\n

c.\u00a0<\/span>Streams API<\/h3>\n

Moreover, to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams, the streams API permits an application. <\/span><\/p>\n

d.\u00a0<\/span>Connector API<\/h3>\n

While it comes to building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems, we use the Connector API. For example, a connector to a relational database might capture every change to a table.<\/span><\/p>\n

Apache Kafka Architecture – Cluster<\/span><\/h2>\n

The below diagram shows the cluster <\/strong>diagram of Apache Kafka:<\/span><\/p>\n

\"Kafka<\/a>

Kafka Architecture – Kafka Cluster<\/p><\/div>\n

Let\u2019s describe each component of Kafka Architecture shown in the above diagram:<\/p>\n

a. Kafka\u00a0Broker<\/h3>\n

Basically, to maintain load balance Kafka cluster<\/strong> typically consists of multiple brokers. However, these are stateless, hence for maintaining the cluster state they use ZooKeeper. Although, one Kafka Broker instance can handle hundreds of thousands of reads and writes per second.<\/p>\n

Whereas, without performance impact, each broker can handle TB of messages. In addition, make sure ZooKeeper performs Kafka broker leader election.<\/p>\n

b. Kafka – ZooKeeper<\/h3>\n

For the purpose of managing and coordinating, Kafka broker uses ZooKeeper<\/strong>. Also, uses it to notify producer and consumer about the presence of any new broker in the Kafka system or failure of the broker in the Kafka system.<\/p>\n

As soon as Zookeeper send the notification regarding presence or failure of the broker then producer and consumer, take the decision and starts coordinating their task with some other broker.<\/p>\n

c. Kafka Producers<\/h3>\n

Further, Producers in Kafka<\/strong> push data to brokers. Also, all the producers search it and automatically sends a message to that new broker, exactly when the new broker starts.<\/p>\n

However, keep in mind that the Kafka producer sends messages as fast as the broker can handle, it doesn\u2019t wait for acknowledgments from the broker.<\/p>\n

d. Kafka Consumers<\/h3>\n

Basically, by using partition offset the Kafka Consumer<\/strong> maintains that how many messages have been consumed because Kafka brokers are stateless. Moreover, you can assure that the consumer has consumed all prior messages once the consumer acknowledges a particular message offset. <\/span><\/p>\n

Also, in order to have a buffer of bytes ready to consume, the consumer issues an asynchronous pull request to the broker. Then simply by supplying an offset value, consumers can rewind or skip to any point in a partition. In addition, ZooKeeper notifies Consumer offset value.<\/span><\/p>\n

Kafka Architecture – Fundamental Concepts<\/span><\/h2>\n

Here, we are listing some of the fundamental concepts of Kafka Architecture that you must know: <\/span><\/p>\n

a. Kafka\u00a0<\/strong>Topics<\/h3>\n

The topic is a logical channel to which producers publish message and from which the consumers receive messages.<\/span><\/p>\n

    \n
  1. A topic defines the stream of a particular type\/classification of data, in Kafka.<\/span><\/li>\n
  2. Moreover, here messages are structured or organized.\u00a0A particular type of messages is published on a particular topic.<\/span><\/li>\n
  3. Basically, at first, a producer writes its messages to the topics. Then consumers read those messages from topics.<\/li>\n
  4. In a Kafka cluster, a topic is identified by its name and must be unique.<\/span><\/li>\n
  5. There can be any number of topics, there is no limitation.<\/span><\/li>\n
  6. We can not change or update data, as soon as it gets published.<\/li>\n<\/ol>\n

    Below is the image which shows the relationship between Kafka Topics and Partitions:<\/p>\n

    \"Kafka<\/a>

    Kafka Architecture – Relation between Kafka Topics and Partitions<\/p><\/div>\n

    b. Partitions in Kafka<\/h3>\n

    In a Kafka cluster, Topics are split into Partitions and also replicated across brokers.<\/span><\/p>\n

      \n
    1. However, to which partition a published message will be written, there is no guarantee about that.<\/span><\/li>\n
    2. Also, we can add a key to a message. Basically, we will get ensured that all these messages (with the same key) will end up in the same partition if a producer publishes a message with a key. Due to this feature, Kafka offers message sequencing guarantee. Though, unless a key is added to it, data is written to partitions randomly. <\/span><\/li>\n
    3. Moreover, in one partition, messages are stored in the sequenced fashion.<\/span><\/li>\n
    4. In a partition, each message is assigned an incremental id, also called offset.<\/span><\/li>\n
    5. However, only within the partition, these offsets are meaningful. Moreover, in a topic, it does not have any value across partitions.<\/span><\/li>\n
    6. There can be any number of Partitions, there is no limitation.<\/li>\n<\/ol>\n

      c. Topic Replication Factor in Kafka<\/span><\/h3>\n

      While designing a Kafka system, it\u2019s always a wise decision to factor in topic replication. As a result, its topics\u2019 replicas from another broker can solve the crisis, if a broker goes down. For example, we have 3 brokers and 3 topics. <\/span><\/p>\n

      Broker1 has Topic 1 and Partition 0, its replica is in Broker2, so on and so forth. It has got a replication factor of 2; it means it will have one additional copy other than the primary one. Below is the image of\u00a0Topic Replication Factor:<\/span><\/p>\n

      \"Kafka<\/a>

      Kafka Architecture – Topic Replication Factor<\/p><\/div>\n

      Some key points \u2013<\/p>\n

        \n
      1. Replication takes place in the partition level only.<\/span><\/li>\n
      2. For a given partition, only one broker can be a leader, at a time. Meanwhile, other brokers will have in-sync replica; what we call ISR.<\/span><\/li>\n
      3. It is not possible to have the number of replication factor more than the number of available brokers.<\/li>\n<\/ol>\n

        d. Consumer Group<\/h3>\n
          \n
        1. It can have multiple consumer process\/instance running.<\/span><\/li>\n
        2. Basically, one consumer group will have one unique group-id.<\/span><\/li>\n
        3. Moreover,\u00a0exactly one consumer instance reads the data from one partition in one consumer group, at the time of reading.<\/li>\n
        4. Since, there is more than one consumer group, in that case, one instance from each of these groups can read from one single partition.<\/span><\/li>\n
        5. However, there will be some inactive consumers, if the number of consumers exceeds the number of partitions. Let\u2019s understand it with an example if there are 8 consumers and 6 partitions in a single consumer group, that means there will be 2 inactive consumers.<\/span><\/li>\n<\/ol>\n

          So, this was all about Apache Kafka Architecture. Hope you like our explanation.<\/p>\n

          Summary<\/span><\/h2>\n

          We have seen the concept of Kafka Architecture. Moreover, we discussed Kafka components and basic concept. Also, we saw a brief pf Kafka Broker, Consumer, Producer. <\/span><\/p>\n

          Along with this, we discussed Kafka Architecture API. Furthermore, for any query regarding Architecture of Kafka, feel free to ask in the comment section.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

          In our last Kafka Tutorial, we discussed Kafka Use Cases and Applications. Today, in this Kafka Tutorial, we will discuss Kafka Architecture. In this Kafka Architecture article, we will see API’s in Kafka. Moreover,...<\/p>\n","protected":false},"author":5,"featured_media":14671,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[7848,7853,7860,7861,7871,7918,16544,7975,9431,14859,14860],"class_list":["post-14633","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","tag-kafka-architecture","tag-kafka-broker","tag-kafka-components","tag-kafka-concepts","tag-kafka-consumer","tag-kafka-producer","tag-kafka-working","tag-kafka-zookeeper","tag-partitions","tag-topic-replication","tag-topics"],"yoast_head":"\nKafka Architecture and Its Fundamental Concepts - DataFlair<\/title>\n<meta name=\"description\" content=\"Kafka Architecture, Kafka Component,Kafka concepts,Kafka consumer,Kafka producer, Kafka zookeeper, Kafka broker, Topics,Kafka Working, Architecture of Kafka\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/kafka-architecture\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Architecture and Its Fundamental Concepts - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Kafka Architecture, Kafka Component,Kafka concepts,Kafka consumer,Kafka producer, Kafka zookeeper, Kafka broker, Topics,Kafka Working, Architecture of Kafka\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/kafka-architecture\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-04-25T15:01:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Architecture-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kafka Architecture and Its Fundamental Concepts - DataFlair","description":"Kafka Architecture, Kafka Component,Kafka concepts,Kafka consumer,Kafka producer, Kafka zookeeper, Kafka broker, Topics,Kafka Working, Architecture of Kafka","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/","og_locale":"en_US","og_type":"article","og_title":"Kafka Architecture and Its Fundamental Concepts - DataFlair","og_description":"Kafka Architecture, Kafka Component,Kafka concepts,Kafka consumer,Kafka producer, Kafka zookeeper, Kafka broker, Topics,Kafka Working, Architecture of Kafka","og_url":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-04-25T15:01:20+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Architecture-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Kafka Architecture and Its Fundamental Concepts","datePublished":"2018-04-25T15:01:20+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/"},"wordCount":1213,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Architecture-01-1.jpg","keywords":["Kafka architecture","Kafka broker","Kafka components","Kafka concepts","Kafka consumer","Kafka producer","Kafka Working","Kafka zookeeper","Partitions","Topic Replication","Topics"],"articleSection":["Apache Kafka Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/kafka-architecture\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/","url":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/","name":"Kafka Architecture and Its Fundamental Concepts - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Architecture-01-1.jpg","datePublished":"2018-04-25T15:01:20+00:00","description":"Kafka Architecture, Kafka Component,Kafka concepts,Kafka consumer,Kafka producer, Kafka zookeeper, Kafka broker, Topics,Kafka Working, Architecture of Kafka","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/kafka-architecture\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Architecture-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Architecture-01-1.jpg","width":1200,"height":628,"caption":"Kafka Architecture"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/kafka-architecture\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Kafka Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/kafka\/"},{"@type":"ListItem","position":3,"name":"Kafka Architecture and Its Fundamental Concepts"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14633","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=14633"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14633\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/14671"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=14633"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=14633"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=14633"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}