Kafka Broker | Command-line Options and Procedure

In this Apache Kafka tutorial, we are going to learn Kafka Broker. Kafka Broker manages the storage of messages in the topic(s). If Apache Kafka has more than one broker, that is what we call a Kafka cluster.

Moreover, in this Kafka Broker Tutorial, we will learn how to start Kafka Broker and Kafka command-line Option.

So, let’s start Apache Kafka Broker.

What is Kafka Broker?

A Kafka broker is also known as Kafka server and a Kafka node. These all names are its synonyms. In simple words, a broker is a mediator between two.

However, Kafka broker is more precisely described as a Message Broker which is responsible for mediating the conversation between different computer systems, guaranteeing delivery of the message to the correct parties.

Hence, the Kafka cluster typically consists of multiple brokers. Kafka Cluster uses Zookeeper for maintaining the cluster state. A single Broker can handle thousands of reads and writes per second.

Whereas, if there is no performance impact, each broker can handle TB of messages. In addition, to be very sure that ZooKeeper performs broker leader election.

Basically, a broker in Kafka is modeled as KafkaServer, which hosts topics. Here, given topics are always partitioned across brokers, in a cluster a single broker hosts topic partitions of one or more topics actually, even when a topic is only partitioned to just a single partition.

In addition, producers send a message to a broker, after receiving broker stores them on disk keyed by unique offset. Moreover, by topic, partition and offset a broker allows consumers to fetch messages.

However, brokers can create a cluster just by sharing information with each other directly or indirectly using Zookeeper. Also, we can say Kafka cluster has exactly one broker which acts as the Controller. By using kafka-server-start.sh script, we can start a single Kafka broker.

How to Start Kafka Broker?

• Start Zookeeper.

./bin/zookeeper-server-start.sh config/zookeeper.properties

We can start a Kafka server, only when Zookeeper is up and running(that will connect to Zookeeper).

./bin/kafka-server-start.sh config/server.properties
kafka-server-start.sh script

Hence, kafka-server-start.sh starts a broker.

$ ./bin/kafka-server-start.sh
USAGE: ./bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*

Note- make sure that Zookeeper is up and running before you run kafka-server-start.sh. Moreover, use zookeeper-server-start shell script.

kafka-server-start.sh uses config/log4j.properties for logging configuration that we can override using KAFKA_LOG4J_OPTS environment variable.

KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:config/log4j.properties"

kafka-server-start.sh accepts KAFKA_HEAP_OPTS and EXTRA_ARGS environment variables.

Kafka Command-line Options

-name

Defaults to kafkaServer when in daemon mode.

-loggc

Enabled when in daemon mode.

-daemon

Enables daemon mode.

–override property=value

A value that should override the value set for property in server.properties file.

$ ./bin/kafka-server-start.sh config/server.properties --override broker.id=100
...
INFO [KafkaServer id=100] started (kafka.server.KafkaServer)

Conclusion

Hence, we have learned the complete Kafka Broker, along with commands to start it. However, if any doubt occurs, feel free to ask in the comment section.