{"id":15347,"date":"2018-05-30T06:24:06","date_gmt":"2018-05-30T06:24:06","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=15347"},"modified":"2018-05-30T06:24:06","modified_gmt":"2018-05-30T06:24:06","slug":"zookeeper-in-kafka","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/zookeeper-in-kafka\/","title":{"rendered":"Role of Apache ZooKeeper in Kafka – Monitoring & Configuration"},"content":{"rendered":"

Today, we will see the Role of Zookeeper in Kafka. This article contains why ZooKeeper is required in Kafka. We can say, ZooKeeper is an inseparable part of Apache Kafka<\/strong>. Before knowing the role of ZooKeeper in Apache Kafka, we will also see what is Apache ZooKeeper. <\/span><\/p>\n

Also, we will cover the introduction to ZooKeeper Production Deployment in detail. Moreover, we are going to discuss how Apache Kafka talks to\u00a0ZooKeeper.<\/span><\/p>\n

In addition, this Kafka ZooKeeper tutorial introduces us with the hardware and configurations for ZooKeeper. Along with this, we are going to study ZooKeeper stable versions and Monitoring in ZooKeeper.<\/span><\/p>\n

So, let’s discuss role of ZooKeeper in Kafka.<\/p>\n

\"role<\/a><\/p>\n

What is ZooKeeper?<\/span><\/h2>\n

Apache ZooKeeper plays the very important role in system architecture as it works in the shadow of more exposed Big Data<\/strong> tools, as Apache Spark<\/strong> or Apache Kafka.\u00a0<\/span><\/p>\n

In other words, Apache Zookeeper is a distributed, open-source configuration, synchronization service along with naming registry for distributed applications.\u00a0<\/span><\/p>\n

\"What<\/a>

What is ZooKeeper<\/p><\/div>\n

Originally, the ZooKeeper framework was built at \u201cYahoo!\u201d. Because it helps to access their applications in an easy manner. Further, for organized service used by Hadoop<\/strong>, HBase<\/strong>, it became a standard and other distributed frameworks.<\/p>\n

Now, let’s discuss the role of\u00a0ZooKeeper in Kafka in detail:<\/p>\n

ZooKeeper in Kafka\u00a0 <\/span><\/h2>\n

Basically, Kafka – ZooKeeper stores a lot of shared information about Kafka Consumers<\/strong> and Kafka Brokers<\/strong>, let’s discuss them in detail:<\/span><\/p>\n

\"Requirement<\/a>

The requirement of ZooKeeper in Kafka<\/p><\/div>\n

a. Kafka Brokers<\/span><\/h3>\n

Below given are the roles of ZooKeeper in\u00a0Kafka Broker<\/strong>:<\/p>\n

i. State<\/strong><\/p>\n

Zookeeper determines the state. That means, it notices, if the Kafka Broker is alive, always when it regularly sends heartbeats requests. Also, while the Broker is the constraint to handle replication, it must be able to follow replication needs.<\/span><\/p>\n

ii. Quotas<\/strong><\/p>\n

In order to have different producing and consuming quotas, Kafka Broker allows some clients. This value is set in ZK under \/config\/clients path. Also, we can change it in bin\/kafka-configs.sh script.<\/span><\/p>\n

iii. Replicas<\/strong><\/p>\n

However, for each topic, Zookeeper in Kafka keeps a set of in-sync replicas (ISR). Moreover, if somehow previously selected leader node fails then on the basis of currently live nodes Apache ZooKeeper will elect the new leader.<\/span><\/p>\n

iv. Nodes and Topics Registry<\/strong><\/p>\n

Basically, Zookeeper in Kafka stores nodes and topic registries. It is possible to find there all available brokers in Kafka and, more precisely, which Kafka topics are held by each broker, under \/brokers\/ids and \/brokers\/topics zNodes, they’re stored. In addition, when it’s started, Kafka broker create the register automatically.<\/span><\/p>\n

b. Kafka Consumers<\/span><\/h3>\n

i. Offsets <\/strong><\/p>\n

ZooKeeper is the default storage engine, for consumer offsets, in Kafka’s 0.9.1 release. However, all information about how many messages Kafka consumer consumes by each consumer is stored in ZooKeeper.<\/span><\/p>\n

ii. Registry<\/strong><\/p>\n

Consumers in Kafka<\/strong> also have their own registry as in the case of Kafka Brokers. However, same rules apply to it, ie. as ephemeral zNode, it’s destroyed once consumer goes down and the registration process is made automatically by the consumer.<\/span><\/p>\n

How does Kafka talk to ZooKeeper?<\/span><\/h2>\n

Here, we will see how Kafka classes are responsible for working with ZooKeeper. <\/span>Scala class <\/strong>representing Kafka is KafkaServer. Its startup() method, initZk() contains a call to method initializing ZooKeeper connection. <\/span><\/p>\n

There are several methods in this algorithm which we use in this Zookeeper method. Hence, as a result, the method creates the temporary connection to ZooKeeper, in this case. <\/span><\/p>\n

This session is responsible for creating zNodes corresponding to chroot if it’s miAfterwarderwards, this connection closes and creates the final connection held by the server.<\/span><\/p>\n

After, still inside initZk(), Kafka initializes all persistent zNodes, especially which server uses. We can retrieve there, among others: \/consumers, \/brokers\/ids, \/brokers\/topics, \/config, \/admin\/delete_topics, \/brokers\/seqid, \/isr_change_notification, \/config\/topics, \/config\/clients.<\/span><\/p>\n

Now, using synchronization to initialize other members, we can use this created ZooKeeper instance:<\/span><\/p>\n