{"id":15076,"date":"2018-05-24T05:45:53","date_gmt":"2018-05-24T05:45:53","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=15076"},"modified":"2018-05-24T05:45:53","modified_gmt":"2018-05-24T05:45:53","slug":"kafka-broker","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-broker\/","title":{"rendered":"Kafka Broker | Command-line Options and Procedure"},"content":{"rendered":"

In this Apache Kafka\u00a0<\/strong>tutorial, we are going to learn Kafka Broker. Kafka Broker manages the storage of messages in the topic(s). If Apache Kafka has more than one broker, that is what we call a Kafka cluster<\/strong>. <\/span><\/p>\n

Moreover, in this Kafka Broker Tutorial, we will learn how to start Kafka Broker and Kafka command-line Option.<\/span><\/p>\n

So, let’s start Apache Kafka Broker.<\/p>\n

What is Kafka Broker?<\/span><\/h2>\n

A Kafka broker is also known as Kafka server and a Kafka node. These all names are its synonyms. In simple words, a broker is a mediator between two. <\/span><\/p>\n

However, Kafka broker is more precisely described as a Message Broker which is responsible for mediating the conversation between different computer systems, guaranteeing delivery of the message to the correct parties.<\/span><\/p>\n

Hence, the Kafka cluster typically consists of multiple brokers. Kafka\u00a0Cluster uses Zookeeper for maintaining the cluster state. A single Broker can handle thousands of reads and writes per second.<\/span><\/p>\n

Whereas, if there is no performance impact, each broker can handle TB of messages. In addition, to be very sure that ZooKeeper performs broker leader election.<\/p>\n

\"Kafka<\/a><\/p>\n

Basically, a broker in Kafka is modeled as KafkaServer, which hosts topics. Here, given topics are always partitioned across brokers, in a cluster a single broker hosts topic partitions of one or more topics actually, even when a topic is only partitioned to just a single partition.<\/span><\/p>\n

In addition, producers<\/strong> send a message to a broker, after receiving broker stores them on disk keyed by unique offset. Moreover, by topic, partition and offset a broker allows consumers to fetch messages. <\/span><\/p>\n

However, brokers can create a cluster just by sharing information with each other directly or indirectly using Zookeeper. Also, we can say Kafka cluster has exactly one broker which acts as the Controller. By using kafka-server-start.sh<\/strong> script, we can start a single Kafka broker.<\/span><\/p>\n

How to Start Kafka Broker?<\/span><\/h2>\n
    \n
  • Start Zookeeper.<\/span><\/li>\n<\/ul>\n
    .\/bin\/zookeeper-server-start.sh config\/zookeeper.properties<\/pre>\n
    We can start a Kafka server, only when Zookeeper is up and running(that will connect to Zookeeper).<\/span><\/p>\n
    .\/bin\/kafka-server-start.sh config\/server.properties\nkafka-server-start.sh script<\/pre>\n
    Hence, <\/span>kafka-server-start.sh<\/b> starts a broker.<\/span><\/p>\n
    $ .\/bin\/kafka-server-start.sh\nUSAGE: .\/bin\/kafka-server-start.sh [-daemon] server.properties [--override property=value]*<\/pre>\n
    Note-<\/strong> make sure that Zookeeper is up and running before you run <\/span>kafka-server-start.sh<\/b>. Moreover, use <\/span>zookeeper-server-start <\/b>shell script.<\/span><\/b><\/p>\n
    kafka-server-start.sh<\/b> uses config\/log4j.properties for logging configuration that we can override using <\/span>KAFKA_LOG4J_OPTS<\/b> environment variable.<\/span><\/p>\n
    KAFKA_LOG4J_OPTS=\"-Dlog4j.configuration=file:config\/log4j.properties\"<\/pre>\n
    kafka-server-start.sh<\/b> accepts <\/span>KAFKA_HEAP_OPTS and EXTRA_ARGS<\/b> environment variables.<\/span><\/p>\n
    Kafka Command-line Options<\/span><\/h2>\n
    -name <\/b><\/p>\n
    Defaults to kafkaServer when in daemon mode.<\/span>
    \n<\/b><\/p>\n
    -loggc<\/b><\/p>\n
    Enabled when in daemon mode.<\/span>
    \n<\/b><\/p>\n
    -daemon<\/b><\/p>\n
    Enables daemon mode.<\/span>
    \n<\/b><\/p>\n
    –override property=value <\/b><\/p>\n
    A value that should override the value set for property in server.properties file.<\/span><\/p>\n
    $ .\/bin\/kafka-server-start.sh config\/server.properties --override broker.id=100\n...\nINFO [KafkaServer id=100] started (kafka.server.KafkaServer)\n<\/pre>\n
    Conclusion<\/h2>\n
    Hence, we have learned the complete Kafka Broker, along with commands to start it. However, if any doubt occurs, feel free to ask in the comment section.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
    In this Apache Kafka\u00a0tutorial, we are going to learn Kafka Broker. Kafka Broker manages the storage of messages in the topic(s). If Apache Kafka has more than one broker, that is what we call...<\/p>\n","protected":false},"author":5,"featured_media":16646,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[825,829,2669,2671,2673,7853,7859,7909,7943],"class_list":["post-15076","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","tag-apache-kafka","tag-apache-kafka-broker","tag-command","tag-command-line","tag-command-line-options","tag-kafka-broker","tag-kafka-command","tag-kafka-node","tag-kafka-server"],"yoast_head":"\nKafka Broker | Command-line Options and Procedure - DataFlair<\/title>\n<meta name=\"description\" content=\"Apache Kafka Broker, Define Broker in Kafka, How to start Kafka Broker, Command-line options in Kafka, Working of Kafka Broker, Kafka tutorial\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/kafka-broker\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Broker | Command-line Options and Procedure - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Apache Kafka Broker, Define Broker in Kafka, How to start Kafka Broker, Command-line options in Kafka, Working of Kafka Broker, Kafka tutorial\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/kafka-broker\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-24T05:45:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Broker-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kafka Broker | Command-line Options and Procedure - DataFlair","description":"Apache Kafka Broker, Define Broker in Kafka, How to start Kafka Broker, Command-line options in Kafka, Working of Kafka Broker, Kafka tutorial","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/kafka-broker\/","og_locale":"en_US","og_type":"article","og_title":"Kafka Broker | Command-line Options and Procedure - DataFlair","og_description":"Apache Kafka Broker, Define Broker in Kafka, How to start Kafka Broker, Command-line options in Kafka, Working of Kafka Broker, Kafka tutorial","og_url":"https:\/\/data-flair.training\/blogs\/kafka-broker\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-05-24T05:45:53+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Broker-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Kafka Broker | Command-line Options and Procedure","datePublished":"2018-05-24T05:45:53+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/"},"wordCount":463,"commentCount":7,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Broker-01.jpg","keywords":["Apache Kafka","Apache Kafka Broker","Command","command line","Command line options","Kafka broker","Kafka Command","Kafka Node","Kafka server"],"articleSection":["Apache Kafka Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/kafka-broker\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/","url":"https:\/\/data-flair.training\/blogs\/kafka-broker\/","name":"Kafka Broker | Command-line Options and Procedure - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Broker-01.jpg","datePublished":"2018-05-24T05:45:53+00:00","description":"Apache Kafka Broker, Define Broker in Kafka, How to start Kafka Broker, Command-line options in Kafka, Working of Kafka Broker, Kafka tutorial","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/kafka-broker\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Broker-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/Kafka-Broker-01.jpg","width":1200,"height":628,"caption":"Apache Kafka Broker"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/kafka-broker\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Kafka Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/kafka\/"},{"@type":"ListItem","position":3,"name":"Kafka Broker | Command-line Options and Procedure"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/15076","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=15076"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/15076\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/16646"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=15076"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=15076"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=15076"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}