{"id":17978,"date":"2018-06-12T04:00:20","date_gmt":"2018-06-12T04:00:20","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=17978"},"modified":"2018-06-12T04:00:20","modified_gmt":"2018-06-12T04:00:20","slug":"kafka-cluster","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/","title":{"rendered":"Way to Setup Kafka Cluster: Apache Kafka"},"content":{"rendered":"

Today, in this Kafka <\/strong>article, we will see Kafka Cluster Setup. This Kafka Cluster tutorial provide us some simple steps to setup Kafka Cluster. In simple words, for high availability of the Kafka service, we need to setup Kafka in cluster mode.\u00a0<\/span><\/p>\n

So, in this Kafka Cluster document, we will learn Kafka multi-node cluster setup and Kafka multi-broker cluster setup. Also, we will\u00a0see Kafka Zookeeper cluster setup.<\/p>\n

So, let’s start Kafka Cluster Setup.<\/p>\n

Kafka Cluster Setup<\/span><\/h2>\n

In order to gain better reliability and high availability of the Kafka service, we need to setup Kafka in cluster mode. At very first:<\/span><\/p>\n

    \n
  • From Apache’s site, download Kafka. Also, extract the zip file. <\/span><\/li>\n
  • Further, make two copies of the extracted folder, and then add the suffix _1, _2, _3 to these folders name. Hence, you will have the folders kafka_2.11-1.1.0_1, kafka_2.11-1.1.0_2, kafka_2.11-1.1.0_3, if our extracted folder name was kafka_2.11-1.1.0. <\/span><\/li>\n
  • Go to the kafka_2.11-1.1.0_1 folder.<\/span><\/li>\n<\/ul>\n

    Steps to Setup Kafka Cluster<\/h2>\n

    Now, follow several steps to set up Kafka Cluster:<\/span><\/p>\n

      \n
    1. Make a folder of name \u201clogs\u201d. In this folder, all the Kafka logs will be stored.<\/span><\/li>\n
    2. Then, open the server.properties file, on going to the config directory. Here, we will find the file, which contains Kafka broker<\/strong> configurations.<\/span><\/li>\n
    3. Further, set broker.id to 1. Make sure it is the id of the broker in a Kafka Cluster, so for each broker, it must be unique.<\/span><\/li>\n
    4. Then, uncomment the listener’s configuration and also set it to PLAINTEXT:\/\/localhost:9091<\/strong>. It says, for connection requests, the Kafka broker will be listening on port 9091.<\/span><\/li>\n
    5. Moreover, with the logs folder path<\/span>, <\/span>set the log.dirs configuration that we created in step 1.<\/span><\/li>\n
    6. Also, set the Apache Zookeeper address, in the zookeeper.connect configuration<\/strong>. However, if Zookeeper is running in a Kafka cluster, then ensure to give the address as a comma-separated list, i.e.:localhost:2181, localhost:2182. <\/span><\/li>\n<\/ol>\n

      Basically, these are some general configurations that we need to be set up for the development environment.<\/span>
      \nIn this way, our first Kafka broker configuration is ready. Now, follow the same steps with the following changes, for the other two folders or brokers.<\/span><\/p>\n

        \n
      • Now, change broker.id to 2 and 3, In step 3, respectively.<\/span><\/li>\n
      • And, also change the ports used to 9092 and 9093, respectively, in step 4.Note: It is possible to provide any port number, which is available.<\/span><\/li>\n<\/ul>\n

        Therefore, for all brokers, our configuration is ready. Now, run the command .\/bin\/kafka-server-start.sh config\/server.properties<\/strong>, on going to the home directory of each Kafka folder. <\/span><\/p>\n

          \n
        • Execute the command (all as one line):<\/span><\/li>\n<\/ul>\n
          .\/bin\/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor\n3 --partitions 50 --topic demo<\/pre>\n
          Here with a replication factor of three for each partition, 50 partitions are created. On defining a replication factor of three, \u00a0there will be one leader and two followers,<\/span> for a partition<\/span>. Also, at the time when message or record is sent to the leader, it is copied in followers.<\/span><\/p>\n
            \n
          • Execute this command:<\/strong><\/li>\n<\/ul>\n
            .\/bin\/kafka-topics.sh --describe --topic Hello-Kafka --zookeeper localhost:2181<\/pre>\n
            It helps us to know that which broker is the leader or follower for which partition. <\/span><\/p>\n
              \n
            • Output:<\/strong><\/li>\n<\/ul>\n
              Topic:demoPartitionCount:50ReplicationFactor:3Configs:<\/b>
              \nTopic: demoPartition: 0Leader: 2Replicas: 2,3,1Isr: 2,3,1<\/b>
              \nTopic: demoPartition: 1Leader: 3Replicas: 3,1,2Isr: 3,1,2<\/b>
              \nTopic: demoPartition: 2Leader: 1Replicas: 1,2,3Isr: 1,2,3<\/b>
              \nTopic: demoPartition: 3Leader: 2Replicas: 2,1,3Isr: 2,1,3<\/b>
              \nTopic: demoPartition: 4Leader: 3Replicas: 3,2,1Isr: 3,2,1<\/b>
              \nTopic: demoPartition: 5Leader: 1Replicas: 1,3,2Isr: 1,3,2<\/b>
              \nTopic: demoPartition: 6Leader: 2Replicas: 2,3,1Isr: 2,3,1<\/b>
              \n…………………………………………………………<\/b>
              \n \u00a0\u00a0\u00a0…………………………………………………………<\/b>
              \n \u00a0\u00a0\u00a0…………………………………………………………<\/b>
              \nNow, we can see Broker 2 is the leader, for Partition 0 and Broker 3 is the leader, for partition 1. And, here ISR refers to in sync replicas.<\/p>\n
              So, this was all about Kafka Cluster. Hope you like our explanation<\/p>\n
              Conclusion<\/span><\/h2>\n
              Hence, in this Kafka Cluster Setup tutorial, we have learned\u00a0the Kafka Cluster Setup of three brokers. Moreover, we discussed Kafka multi-node setup & Kafka – Zookeeper setup. Still, if any doubt regarding Kafka Cluster Setup, ask in the comment tab.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
              Today, in this Kafka article, we will see Kafka Cluster Setup. This Kafka Cluster tutorial provide us some simple steps to setup Kafka Cluster. In simple words, for high availability of the Kafka service,...<\/p>\n","protected":false},"author":5,"featured_media":18184,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[825,7843,7858,7887,7906,7907,7908,7987],"class_list":["post-17978","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","tag-apache-kafka","tag-kafka-3-broker-setup","tag-kafka-cluster","tag-kafka-in-cluster-mode","tag-kafka-multi-broker-setup","tag-kafka-multi-nodes-setup","tag-kafka-multibroker-cluster","tag-kafka-zookeeper-cluster-setup"],"yoast_head":"\nWay to Setup Kafka Cluster: Apache Kafka - DataFlair<\/title>\n<meta name=\"description\" content=\"Apache Kafka Cluster setup tutorial: Kafka-Zookeeper cluster setup,install Apache Kafka multi-nodes & multi-brokers, kafka multibroker cluster,Kafka Cluster\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/kafka-cluster\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Way to Setup Kafka Cluster: Apache Kafka - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Apache Kafka Cluster setup tutorial: Kafka-Zookeeper cluster setup,install Apache Kafka multi-nodes & multi-brokers, kafka multibroker cluster,Kafka Cluster\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/kafka-cluster\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-12T04:00:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Way-to-Setup-Kafka-Cluster-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Way to Setup Kafka Cluster: Apache Kafka - DataFlair","description":"Apache Kafka Cluster setup tutorial: Kafka-Zookeeper cluster setup,install Apache Kafka multi-nodes & multi-brokers, kafka multibroker cluster,Kafka Cluster","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/","og_locale":"en_US","og_type":"article","og_title":"Way to Setup Kafka Cluster: Apache Kafka - DataFlair","og_description":"Apache Kafka Cluster setup tutorial: Kafka-Zookeeper cluster setup,install Apache Kafka multi-nodes & multi-brokers, kafka multibroker cluster,Kafka Cluster","og_url":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-06-12T04:00:20+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Way-to-Setup-Kafka-Cluster-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Way to Setup Kafka Cluster: Apache Kafka","datePublished":"2018-06-12T04:00:20+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/"},"wordCount":615,"commentCount":2,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Way-to-Setup-Kafka-Cluster-01.jpg","keywords":["Apache Kafka","Kafka 3 broker setup","Kafka Cluster","kafka in cluster mode","Kafka Multi-broker setup","Kafka multi-nodes setup","kafka multibroker cluster","Kafka-Zookeeper cluster setup"],"articleSection":["Apache Kafka Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/kafka-cluster\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/","url":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/","name":"Way to Setup Kafka Cluster: Apache Kafka - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Way-to-Setup-Kafka-Cluster-01.jpg","datePublished":"2018-06-12T04:00:20+00:00","description":"Apache Kafka Cluster setup tutorial: Kafka-Zookeeper cluster setup,install Apache Kafka multi-nodes & multi-brokers, kafka multibroker cluster,Kafka Cluster","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/kafka-cluster\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Way-to-Setup-Kafka-Cluster-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Way-to-Setup-Kafka-Cluster-01.jpg","width":1200,"height":628,"caption":"Way to Setup Kafka Cluster: Apache Kafka"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/kafka-cluster\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Kafka Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/kafka\/"},{"@type":"ListItem","position":3,"name":"Way to Setup Kafka Cluster: Apache Kafka"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/17978","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=17978"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/17978\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/18184"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=17978"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=17978"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=17978"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}