{"id":21521,"date":"2018-07-16T08:28:58","date_gmt":"2018-07-16T08:28:58","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=21521"},"modified":"2018-07-16T08:28:58","modified_gmt":"2018-07-16T08:28:58","slug":"kafka-topic-architecture","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/","title":{"rendered":"Apache Kafka Topic – Architecture & Partitions"},"content":{"rendered":"

In this Kafka<\/strong> article, we will learn the whole concept of a Kafka Topic along with Kafka Architecture. Where architecture in Kafka<\/strong> includes replication, Failover as well as Parallel Processing. <\/span><\/p>\n

In addition, we will also see the way to create a Kafka topic and example of Apache Kafka Topic to understand Kafka well. Moreover, we will see Kafka partitioning and Kafka log partitioning.<\/span><\/p>\n

So, let\u2019s begin with the Kafka Topic.<\/span><\/p>\n

What is Kafka Topic?<\/span><\/h2>\n

Simply put, a named stream of records is what we call Kafka Topic. Basically, in logs Kafka stores topics. However, a topic log in Apache Kafka is broken up into several partitions. <\/span><\/p>\n

And, further, Kafka spreads those log\u2019s partitions across multiple servers or disks. In other words, we can say a topic in Kafka is a category, stream name or a feed.<\/span><\/p>\n

\"Kafka<\/a>

What is the Topic in Kafka<\/p><\/div>\n

In addition, we can say Topics in Apache Kafka are inherently published as well as subscribe style messaging. <\/span><\/p>\n

Moreover, there can be zero or many subscribers called Kafka Consumer Groups<\/strong> in a Kafka Topic. Basically, these Topics in Kafka are broken up into partitions for speed, scalability, as well as size.<\/span><\/p>\n

How to Create a Kafka Topic<\/span><\/h2>\n

At very first, run kafka-topics.sh and specify the topic name, replication factor, and other attributes, to create a topic in Kafka:<\/span><\/p>\n

\/bin\/kafka-topics.sh --create \\\n   --zookeeper <hostname>:<port> \\\n   --topic <topic-name> \\\n   --partitions <number-of-partitions> \\\n   --replication-factor <number-of-replicating-servers><\/pre>\n
Now, with one partition and one replica, below example creates a topic named “test1”:<\/span><\/p>\n
bin\/kafka-topics.sh --create \\\n   --zookeeper localhost:2181 \\\n   --replication-factor 1 \\\n   --partitions 1 \\\n   --topic text<\/pre>\n
Further, run the list topic command, to view the topic:<\/span><\/p>\n
> bin\/kafka-topics.sh --list --zookeeper localhost:2181\ntest1<\/pre>\n
Make sure, when applications attempt to produce, consume, or fetch metadata for a nonexistent topic, the auto.create.topics.enable property, when set to true, automatically creates topics.<\/span><\/p>\n
Kafka Topic Partitions<\/span><\/h2>\n
Further, Kafka breaks topic logs up into several partitions. So, usually by record key if the key is present and round-robin, a record is stored on a partition while the key is missing (default behavior).<\/span><\/p>\n
 By default, the key which helps to determines that which partition a Kafka Producer<\/strong> sends the record is the Record Key.<\/span><\/p>\n
Basically, to scale a topic across many servers for producer writes, Kafka uses partitions. Also, in order to facilitate parallel consumers, Kafka uses partitions. Moreover, while it comes to failover, Kafka can replicate partitions to multiple Kafka Brokers<\/strong>.<\/span><\/p>\n
Kafka Topic Log Partition\u2019s Ordering and Cardinality<\/span><\/h2>\n
Well, we can say, only in a single partition, Kafka does maintain record order. As a partition is also an ordered, immutable record sequence. And, by using the partition as a structured commit log, Kafka continually appended to partitions. <\/span><\/p>\n
In partitions, all records are assigned one sequential id number which we further call an offset. That offset further identifies each record location within the partition.<\/span><\/p>\n
In addition, in order to scale beyond a size that will fit on a single server, Topic partitions permits to Kafka log. As topics can span many partitions hosted on many servers but Topic partitions must fit on servers which host it. Moreover, topic partitions in Apache Kafka are a unit of parallelism. <\/span><\/p>\n
That says, at a time, a partition can only be worked on by one Kafka Consumer<\/strong> in a consumer group. Basically, a Consumer in Kafka can only run in their own process or their own thread. Although, Kafka spreads partitions across the remaining consumer in the same consumer group, if a consumer stops.<\/span><\/p>\n
Kafka Topic Partition Replication<\/span><\/h2>\n
For the purpose of fault tolerance, Kafka can perform replication of partitions across a configurable number of Kafka servers. Basically, there is a leader server and zero or more follower servers in each partition. Also, for a partition, leaders are those who handle all read and write requests.<\/span><\/p>\n
However, if the leader dies, the followers replicate leaders and take over. Additionally, for parallel consumer handling within a group, Kafka uses also uses partitions.<\/span><\/p>\n
Replication: Kafka Partition Leaders, Followers, and ISRs.<\/span><\/h2>\n
However, by using ZooKeeper<\/strong>, Kafka chooses one broker\u2019s partition\u2019s replicas as the leader. Also, we can say, for the partition, the broker which has the partition leader handles all reads and writes of records. <\/span><\/p>\n
Moreover, to the leader partition to followers (node\/partition pair), Kafka replicates writes. On defining the term ISR, a follower which is in-sync is what we call an ISR (in-sync replica). Although, Kafka chooses a new ISR<\/em> as the new leader if a partition leader fails.<\/span><\/p>\n
Kafka Architecture: Kafka Replication – Replicating to Partition 0<\/h2>\n
\"Kafka<\/a>
Kafka Architecture: Kafka Replication – Replicating to Partition 0<\/p><\/div>\n
Although, when all ISRs for partition wrote to their log, the record is considered \u201ccommitted\u201d. However, we can only read the committed records from the consumer.<\/p>\n
\"Kafka<\/a>
Kafka Topic – Replication to Partition 1<\/p><\/div>\n
So, this was all about Kafka Topic. Hope you like our explanation.<\/p>\n
Conclusion<\/span><\/h2>\n
Hence, we have seen the whole concept of Kafka Topic in detail. Moreover, we discussed Kafka Topic partitions, log partitions in Kafka Topic, and Kafka replication factor. <\/span><\/p>\n
Also, we saw Kafka Architecture and creating a Topic in Kafka. Still, if any doubt occurs regarding Topics in Kafka, feel free to ask in the comment section. We are happy to help.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
In this Kafka article, we will learn the whole concept of a Kafka Topic along with Kafka Architecture. Where architecture in Kafka includes replication, Failover as well as Parallel Processing. In addition, we will...<\/p>\n","protected":false},"author":5,"featured_media":21673,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[16525,1083,3101,7848,7875,7897,7961,7963,7964,7965,7966,7968,15805,16032],"class_list":["post-21521","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","tag-apache-kafka-topic","tag-architecture-of-kafka","tag-create-topic-in-kafka","tag-kafka-architecture","tag-kafka-consumer-group","tag-kafka-log-partitions","tag-kafka-topic","tag-kafka-topic-example","tag-kafka-topic-partitions","tag-kafka-topic-replication","tag-kafka-topic-tutorial","tag-kafka-tutorial","tag-what-is-kafka-topic","tag-what-is-topic-in-kafka"],"yoast_head":"\nApache Kafka Topic - Architecture & Partitions - DataFlair<\/title>\n<meta name=\"description\" content=\"Kafka Topic tutorial,what is Topic in Kafka,Kafka Consumer Group,Kafka Architecture,Kafka partition,Kafka Topic Example,Kafka replication factor,Kafka log\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Kafka Topic - Architecture & Partitions - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Kafka Topic tutorial,what is Topic in Kafka,Kafka Consumer Group,Kafka Architecture,Kafka partition,Kafka Topic Example,Kafka replication factor,Kafka log\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-16T08:28:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Apache-Kafka-Topic-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Kafka Topic - Architecture & Partitions - DataFlair","description":"Kafka Topic tutorial,what is Topic in Kafka,Kafka Consumer Group,Kafka Architecture,Kafka partition,Kafka Topic Example,Kafka replication factor,Kafka log","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/","og_locale":"en_US","og_type":"article","og_title":"Apache Kafka Topic - Architecture & Partitions - DataFlair","og_description":"Kafka Topic tutorial,what is Topic in Kafka,Kafka Consumer Group,Kafka Architecture,Kafka partition,Kafka Topic Example,Kafka replication factor,Kafka log","og_url":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-07-16T08:28:58+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Apache-Kafka-Topic-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Apache Kafka Topic – Architecture & Partitions","datePublished":"2018-07-16T08:28:58+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/"},"wordCount":866,"commentCount":4,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Apache-Kafka-Topic-01.jpg","keywords":["Apache Kafka topic","architecture of kafka","create topic in kafka","Kafka architecture","Kafka Consumer Group","Kafka Log Partitions","kafka Topic","Kafka Topic example","Kafka Topic partitions","Kafka Topic replication","Kafka Topic tutorial","Kafka tutorial","what is Kafka Topic","What is topic in Kafka"],"articleSection":["Apache Kafka Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/","url":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/","name":"Apache Kafka Topic - Architecture & Partitions - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Apache-Kafka-Topic-01.jpg","datePublished":"2018-07-16T08:28:58+00:00","description":"Kafka Topic tutorial,what is Topic in Kafka,Kafka Consumer Group,Kafka Architecture,Kafka partition,Kafka Topic Example,Kafka replication factor,Kafka log","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Apache-Kafka-Topic-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Apache-Kafka-Topic-01.jpg","width":1200,"height":628,"caption":"Apache Kafka Topic - Architecture & Partitions"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/kafka-topic-architecture\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Kafka Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/kafka\/"},{"@type":"ListItem","position":3,"name":"Apache Kafka Topic – Architecture & Partitions"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/21521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=21521"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/21521\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/21673"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=21521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=21521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=21521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}