Way to Setup Kafka Cluster: Apache Kafka

Today, in this Kafka article, we will see Kafka Cluster Setup. This Kafka Cluster tutorial provide us some simple steps to setup Kafka Cluster. In simple words, for high availability of the Kafka service, we need to setup Kafka in cluster mode. 

So, in this Kafka Cluster document, we will learn Kafka multi-node cluster setup and Kafka multi-broker cluster setup. Also, we will see Kafka Zookeeper cluster setup.

So, let’s start Kafka Cluster Setup.

Kafka Cluster Setup

In order to gain better reliability and high availability of the Kafka service, we need to setup Kafka in cluster mode. At very first:

• From Apache’s site, download Kafka. Also, extract the zip file.
• Further, make two copies of the extracted folder, and then add the suffix _1, _2, _3 to these folders name. Hence, you will have the folders kafka_2.11-1.1.0_1, kafka_2.11-1.1.0_2, kafka_2.11-1.1.0_3, if our extracted folder name was kafka_2.11-1.1.0.
• Go to the kafka_2.11-1.1.0_1 folder.

Steps to Setup Kafka Cluster

Now, follow several steps to set up Kafka Cluster:

1. Make a folder of name “logs”. In this folder, all the Kafka logs will be stored.
2. Then, open the server.properties file, on going to the config directory. Here, we will find the file, which contains Kafka broker configurations.
3. Further, set broker.id to 1. Make sure it is the id of the broker in a Kafka Cluster, so for each broker, it must be unique.
4. Then, uncomment the listener’s configuration and also set it to PLAINTEXT://localhost:9091. It says, for connection requests, the Kafka broker will be listening on port 9091.
5. Moreover, with the logs folder path, set the log.dirs configuration that we created in step 1.
6. Also, set the Apache Zookeeper address, in the zookeeper.connect configuration. However, if Zookeeper is running in a Kafka cluster, then ensure to give the address as a comma-separated list, i.e.:localhost:2181, localhost:2182.

Basically, these are some general configurations that we need to be set up for the development environment.
In this way, our first Kafka broker configuration is ready. Now, follow the same steps with the following changes, for the other two folders or brokers.

• Now, change broker.id to 2 and 3, In step 3, respectively.
• And, also change the ports used to 9092 and 9093, respectively, in step 4.Note: It is possible to provide any port number, which is available.

Therefore, for all brokers, our configuration is ready. Now, run the command ./bin/kafka-server-start.sh config/server.properties, on going to the home directory of each Kafka folder.

• Execute the command (all as one line):

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor
3 --partitions 50 --topic demo

Here with a replication factor of three for each partition, 50 partitions are created. On defining a replication factor of three,  there will be one leader and two followers, for a partition. Also, at the time when message or record is sent to the leader, it is copied in followers.

• Execute this command:

./bin/kafka-topics.sh --describe --topic Hello-Kafka --zookeeper localhost:2181

It helps us to know that which broker is the leader or follower for which partition.

• Output:

Topic:demoPartitionCount:50ReplicationFactor:3Configs:
Topic: demoPartition: 0Leader: 2Replicas: 2,3,1Isr: 2,3,1
Topic: demoPartition: 1Leader: 3Replicas: 3,1,2Isr: 3,1,2
Topic: demoPartition: 2Leader: 1Replicas: 1,2,3Isr: 1,2,3
Topic: demoPartition: 3Leader: 2Replicas: 2,1,3Isr: 2,1,3
Topic: demoPartition: 4Leader: 3Replicas: 3,2,1Isr: 3,2,1
Topic: demoPartition: 5Leader: 1Replicas: 1,3,2Isr: 1,3,2
Topic: demoPartition: 6Leader: 2Replicas: 2,3,1Isr: 2,3,1
…………………………………………………………
   …………………………………………………………
   …………………………………………………………
Now, we can see Broker 2 is the leader, for Partition 0 and Broker 3 is the leader, for partition 1. And, here ISR refers to in sync replicas.

So, this was all about Kafka Cluster. Hope you like our explanation

Conclusion

Hence, in this Kafka Cluster Setup tutorial, we have learned the Kafka Cluster Setup of three brokers. Moreover, we discussed Kafka multi-node setup & Kafka – Zookeeper setup. Still, if any doubt regarding Kafka Cluster Setup, ask in the comment tab.