So, let’s continue the tutorial.<\/p>\n
What is Kafka?<\/span><\/h3>\nApache Kafka is a fast, scalable, fault-tolerant messaging system which enables communication between producers and consumers using message-based topics. In simple words, it designs a platform for high-end new generation distributed applications.<\/p>\n
Also, it allows a large number of permanent or ad-hoc consumers. One of the best features of Kafka is, it is highly available and resilient to node failures and supports automatic recovery. <\/span><\/p>\nThis feature makes Apache Kafka ideal for communication and integration between components of large-scale data systems in real-world data systems. <\/span><\/p>\nMoreover, this technology takes place of conventional message brokers like JMS, AMQP with the ability to give higher throughput, reliability, and replication.<\/p>\n
Before moving forward in Kafka Tutorial, let\u2019s understand the Messaging System in Kafka.<\/p>\n
Messaging Systems in Kafka<\/h4>\n
The main task of managing system is to transfer data from one application to another so that the applications can mainly work on data without worrying about sharing it.<\/p>\n
Distributed messaging is based on the reliable message queuing process. Messages are queued non-synchronously between the messaging system and client applications.<\/p>\n
There are two types of messaging patterns available:<\/p>\n
\n- Point to point messaging system<\/li>\n
- Publish-subscribe messaging system<\/li>\n<\/ul>\n
You must check the concept of\u00a0Apache Kafka Queuing<\/strong><\/p>\n![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Kafka-Messaging-System-01.jpg\")
<\/a><\/p>\n\n- Point to Point Messaging System<\/strong><\/li>\n<\/ul>\n
In this messaging system, messages continue to remain in a queue. More than one consumer can consume the messages in the queue but only one consumer can consume a particular message.<\/p>\n
After the consumer reads the message in the queue, the message disappears from that queue.<\/p>\n
\n- Publish-Subscribe Messaging System<\/strong><\/li>\n<\/ul>\n
In this messaging system, messages continue to remain in a Topic. Contrary to Point to point messaging system, consumers can take more than one topic and consume every message in that topic. Message producers are known as publishers and Kafka consumers<\/strong> are known as subscribers.<\/p>\nHistory of Apache Kafka<\/h4>\n
Previously, LinkedIn was facing the issue of low latency ingestion of huge amount of data from the website into a lambda architecture which could be able to process real-time events.<\/span><\/p>\n As a solution, Apache Kafka was developed in the year 2010, since none of the solutions was available to deal with this drawback, before.<\/span><\/p>\nHowever, there were technologies available for batch processing, but the deployment details of those technologies were shared with the downstream users. Hence, while it comes to Real-time Processing, those technologies were not enough suitable. Then, in the year 2011 Kafka was made public.<\/p>\n
Need of Apache Kafka Cluster<\/h4>\n
As we all know, there is an enormous volume of data used in Big Data. For data, there are two main challenges. One is how to collect and manage a large volume of data and the other one is the analysis of the collected data. For dealing with such challenges, you need to have a messaging system.<\/p>\n
There are many benefits of Apache Kafka that justifies the usage of Apache Kafka:<\/p>\n
\n- Tracking web activities by storing\/sending the events for real-time processes.<\/span><\/li>\n
- Alerting and reporting the operational metrics.<\/span><\/li>\n
- Transforming data into the standard format.<\/span><\/li>\n
- Continuous processing of streaming data to the topics.<\/span><\/li>\n<\/ul>\n
Explore the benefits and limitations of Apache Kafka in detail.<\/strong><\/p>\nTherefore, this technology is giving a tough competition to some of the most popular applications like ActiveMQ, RabbitMQ, AWS, etc because of its wide use.<\/p>\n
Audience for Kafka Tutorial<\/h4>\n
Professionals who are aspiring to make a career in Big Data Analytics using Apache Kafka messaging system should refer to this Kafka Tutorial. It will give you a complete understanding of Apache Kafka.<\/span><\/p>\nPrerequisites to Kafka<\/h4>\n
You must have a good understanding of Java, Scala, Distributed messaging system, and Linux environment, before proceeding with this Apache Kafka Tutorial.<\/span><\/p>\nKafka Architecture<\/h3>\n
Below we are discussing four core APIs in this Apache Kafka tutorial:<\/p>\n
![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Apache-Kafka-Architecture.png\")
<\/a><\/p>\n1. Kafka Producer API<\/strong><\/p>\nThis Kafka Producer API permits an application to publish a stream of records to one or more Kafka topics.<\/span><\/p>\n2.\u00a0Kafka Consumer API<\/strong><\/p>\nThe Consumer API permits an application to take one or more topics and process the continous flow of records produced to them.<\/p>\n
3.\u00a0Kafka Streams API<\/strong><\/p>\nThe Streams API permits an application to behave as a stream processor, consuming an input stream from one or more topics and generating an output stream to one or more output topics, efficiently modifying the input streams to output streams.<\/p>\n
4.\u00a0Kafka Connector API<\/strong><\/p>\nThe Connector API permits creating and running reusable producers or consumers that enables connection between Kafka topics and existing applications or data systems.<\/b><\/p>\nKafka Components<\/h3>\n
Using the following components, Kafka achieves messaging:<\/p>\n
1. Kafka Topic<\/h3>\n
A bunch of messages that belong to a particular category is known as a Topic. Data stores in topics. In addition, we can replicate and partition Topics. Here, replicate refers to copies and partition refers to the division. Also, visualize them as logs wherein, Kafka stores messages. However, this ability to replicate and partitioning topics is one of the factors that enable Kafka’s fault tolerance and scalability.<\/span><\/p>\n![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Kafka-Topic-1.png\")
<\/a><\/p>\n2. Kafka Producer<\/h3>\n
The producers publish the messages on one or more Kafka topics.<\/p>\n
3.\u00a0Kafka Consumer<\/h3>\n
Consumers take one or more topics and consume messages that are already published through extracting data from the brokers.<\/p>\n
4.\u00a0Kafka Broker<\/h3>\n
These are basically systems which maintain the published data. A single broker can have zero or more partitions per topic.<\/p>\n
5.\u00a0Kafka Zookeeper<\/h3>\n
With the help of zookeeper, Kafka provides the brokers with metadata regarding the processes running in the system and grants health checking and broker leadership election.<\/p>\n
Log Anatomy<\/h4>\n
We view log as the partitions. Basically, a data source writes messages to the log. One of the advantages is, at any time one or more consumers can read from the log they select. Here, the below diagram shows a log is being written by the data source and read by consumers at different offsets.<\/span><\/p>\n![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Log-Anatomy.png\")
<\/a><\/p>\nData Log<\/span><\/h4>\nBy Kafka, messages are retained for a considerable amount of time. Also, consumers can read as per their convenience. However, if Kafka is configured to keep messages for 24 hours and a consumer is down for greater than 24 hours, the consumer will lose messages. <\/span><\/p>\nAnd, messages can be read from last known offset, if the downtime on part of the consumer is just 60 minutes. Kafka doesn\u2019t keep a track on what consumers are reading from a topic.<\/span><\/p>\nPartition in Kafka<\/span><\/h4>\nThere are few partitions in every Kafka broker. Each partition can be either a leader or a replica of a topic. In addition, along with updating of replicas with new data, Leader is responsible for all writes and reads to a topic. The replica takes over as the new leader if somehow the leader fails.<\/p>\n
![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/04\/Partitioning-in-Kafka-1.png\")
<\/a><\/p>\nImportance of Java in Apache Kafka<\/span><\/h4>\nApache Kafka is written in pure Java and also Kafka\u2019s native API is java. However, many other languages like C++, Python,\u00a0.Net, Go, etc. also support Kafka. <\/span><\/p>\nStill, a platform where there is no need of using a third-party library is Java. Also, we can say, writing code in languages apart from Java will be a little overhead.<\/span><\/p>\nIn addition, we can use\u00a0<\/strong>Java\u00a0<\/strong>language if we need the high processing rates that come standard on Kafka. Also, Java provides good community support for Kafka consumer clients. Hence, it is the right choice to implement Kafka in Java.<\/p>\nKafka Use Cases<\/span><\/h3>\nThere are several\u00a0use cases of Kafka<\/strong> that show why we actually use Apache Kafka.<\/span><\/p>\n1. Messaging<\/strong><\/p>\nKafka is the best substitute for traditional message brokers. We can say Kafka has better throughput, replication, fault-tolerance\u00a0and built-in partitioning which makes it a better solution for large-scale message processing applications.<\/span><\/p>\n2. Metrics<\/strong><\/p>\nKafka is mostly utilized for operational monitoring data. It includes aggregating statistics from distributed applications to generate centralized feeds of operational data.<\/span><\/p>\n3. Event Sourcing<\/strong><\/p>\nKafka is a great backend for applications of event sourcing since it supports very large stored log data.<\/span><\/p>\nComparisons in Kafka<\/span><\/h3>\nMany applications offer the same functionality as Kafka like ActiveMQ, RabbitMQ, Apache Flume<\/strong>, Storm, and Spark. Then why should you go for Apache Kafka instead of others?<\/p>\nLet\u2019s see the comparisons below:<\/p>\n
1. Apache Kafka vs Apache Flume<\/h4>\n
![\"Kafka](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Apache-Kafka-vs-Apache-Flume-01.jpg\")
<\/a><\/p>\ni. \u00a0Types of tool<\/strong><\/em>
\n<\/b><\/p>\nApache Kafka\u00a0<\/b>–\u00a0It is a general-purpose tool for various producers and consumers.<\/span>
\n<\/b><\/p>\nApache Flume\u00a0<\/b>– Whereas, it is a special-purpose tool for particular applications.<\/span><\/p>\nii. Replication feature<\/strong><\/em>
\n<\/b><\/p>\nApache Kafka\u00a0<\/b>– Using ingest pipelines, it replicates the events.<\/span>
\n<\/b><\/p>\nApache Flume <\/b>– It does not replicate the events.<\/span><\/p>\n2. RabbitMQ vs Apache Kafka<\/h4>\n
This feature makes Apache Kafka ideal for communication and integration between components of large-scale data systems in real-world data systems. <\/span><\/p>\n Moreover, this technology takes place of conventional message brokers like JMS, AMQP with the ability to give higher throughput, reliability, and replication.<\/p>\n Before moving forward in Kafka Tutorial, let\u2019s understand the Messaging System in Kafka.<\/p>\n The main task of managing system is to transfer data from one application to another so that the applications can mainly work on data without worrying about sharing it.<\/p>\n Distributed messaging is based on the reliable message queuing process. Messages are queued non-synchronously between the messaging system and client applications.<\/p>\n There are two types of messaging patterns available:<\/p>\n You must check the concept of\u00a0Apache Kafka Queuing<\/strong><\/p>\n In this messaging system, messages continue to remain in a queue. More than one consumer can consume the messages in the queue but only one consumer can consume a particular message.<\/p>\n After the consumer reads the message in the queue, the message disappears from that queue.<\/p>\n In this messaging system, messages continue to remain in a Topic. Contrary to Point to point messaging system, consumers can take more than one topic and consume every message in that topic. Message producers are known as publishers and Kafka consumers<\/strong> are known as subscribers.<\/p>\n Previously, LinkedIn was facing the issue of low latency ingestion of huge amount of data from the website into a lambda architecture which could be able to process real-time events.<\/span><\/p>\n As a solution, Apache Kafka was developed in the year 2010, since none of the solutions was available to deal with this drawback, before.<\/span><\/p>\n However, there were technologies available for batch processing, but the deployment details of those technologies were shared with the downstream users. Hence, while it comes to Real-time Processing, those technologies were not enough suitable. Then, in the year 2011 Kafka was made public.<\/p>\n As we all know, there is an enormous volume of data used in Big Data. For data, there are two main challenges. One is how to collect and manage a large volume of data and the other one is the analysis of the collected data. For dealing with such challenges, you need to have a messaging system.<\/p>\n There are many benefits of Apache Kafka that justifies the usage of Apache Kafka:<\/p>\n Explore the benefits and limitations of Apache Kafka in detail.<\/strong><\/p>\n Therefore, this technology is giving a tough competition to some of the most popular applications like ActiveMQ, RabbitMQ, AWS, etc because of its wide use.<\/p>\n Professionals who are aspiring to make a career in Big Data Analytics using Apache Kafka messaging system should refer to this Kafka Tutorial. It will give you a complete understanding of Apache Kafka.<\/span><\/p>\n You must have a good understanding of Java, Scala, Distributed messaging system, and Linux environment, before proceeding with this Apache Kafka Tutorial.<\/span><\/p>\n Below we are discussing four core APIs in this Apache Kafka tutorial:<\/p>\n 1. Kafka Producer API<\/strong><\/p>\n This Kafka Producer API permits an application to publish a stream of records to one or more Kafka topics.<\/span><\/p>\n 2.\u00a0Kafka Consumer API<\/strong><\/p>\n The Consumer API permits an application to take one or more topics and process the continous flow of records produced to them.<\/p>\n 3.\u00a0Kafka Streams API<\/strong><\/p>\n The Streams API permits an application to behave as a stream processor, consuming an input stream from one or more topics and generating an output stream to one or more output topics, efficiently modifying the input streams to output streams.<\/p>\n 4.\u00a0Kafka Connector API<\/strong><\/p>\n The Connector API permits creating and running reusable producers or consumers that enables connection between Kafka topics and existing applications or data systems.<\/b><\/p>\n Using the following components, Kafka achieves messaging:<\/p>\n A bunch of messages that belong to a particular category is known as a Topic. Data stores in topics. In addition, we can replicate and partition Topics. Here, replicate refers to copies and partition refers to the division. Also, visualize them as logs wherein, Kafka stores messages. However, this ability to replicate and partitioning topics is one of the factors that enable Kafka’s fault tolerance and scalability.<\/span><\/p>\n The producers publish the messages on one or more Kafka topics.<\/p>\n Consumers take one or more topics and consume messages that are already published through extracting data from the brokers.<\/p>\n These are basically systems which maintain the published data. A single broker can have zero or more partitions per topic.<\/p>\n With the help of zookeeper, Kafka provides the brokers with metadata regarding the processes running in the system and grants health checking and broker leadership election.<\/p>\n We view log as the partitions. Basically, a data source writes messages to the log. One of the advantages is, at any time one or more consumers can read from the log they select. Here, the below diagram shows a log is being written by the data source and read by consumers at different offsets.<\/span><\/p>\n By Kafka, messages are retained for a considerable amount of time. Also, consumers can read as per their convenience. However, if Kafka is configured to keep messages for 24 hours and a consumer is down for greater than 24 hours, the consumer will lose messages. <\/span><\/p>\n And, messages can be read from last known offset, if the downtime on part of the consumer is just 60 minutes. Kafka doesn\u2019t keep a track on what consumers are reading from a topic.<\/span><\/p>\n There are few partitions in every Kafka broker. Each partition can be either a leader or a replica of a topic. In addition, along with updating of replicas with new data, Leader is responsible for all writes and reads to a topic. The replica takes over as the new leader if somehow the leader fails.<\/p>\n Apache Kafka is written in pure Java and also Kafka\u2019s native API is java. However, many other languages like C++, Python,\u00a0.Net, Go, etc. also support Kafka. <\/span><\/p>\n Still, a platform where there is no need of using a third-party library is Java. Also, we can say, writing code in languages apart from Java will be a little overhead.<\/span><\/p>\n In addition, we can use\u00a0<\/strong>Java\u00a0<\/strong>language if we need the high processing rates that come standard on Kafka. Also, Java provides good community support for Kafka consumer clients. Hence, it is the right choice to implement Kafka in Java.<\/p>\n There are several\u00a0use cases of Kafka<\/strong> that show why we actually use Apache Kafka.<\/span><\/p>\n 1. Messaging<\/strong><\/p>\n Kafka is the best substitute for traditional message brokers. We can say Kafka has better throughput, replication, fault-tolerance\u00a0and built-in partitioning which makes it a better solution for large-scale message processing applications.<\/span><\/p>\n 2. Metrics<\/strong><\/p>\n Kafka is mostly utilized for operational monitoring data. It includes aggregating statistics from distributed applications to generate centralized feeds of operational data.<\/span><\/p>\n 3. Event Sourcing<\/strong><\/p>\n Kafka is a great backend for applications of event sourcing since it supports very large stored log data.<\/span><\/p>\n Many applications offer the same functionality as Kafka like ActiveMQ, RabbitMQ, Apache Flume<\/strong>, Storm, and Spark. Then why should you go for Apache Kafka instead of others?<\/p>\n Let\u2019s see the comparisons below:<\/p>\n i. \u00a0Types of tool<\/strong><\/em> Apache Kafka\u00a0<\/b>–\u00a0It is a general-purpose tool for various producers and consumers.<\/span> Apache Flume\u00a0<\/b>– Whereas, it is a special-purpose tool for particular applications.<\/span><\/p>\n ii. Replication feature<\/strong><\/em> Apache Kafka\u00a0<\/b>– Using ingest pipelines, it replicates the events.<\/span> Apache Flume <\/b>– It does not replicate the events.<\/span><\/p>\nMessaging Systems in Kafka<\/h4>\n
\n
<\/a><\/p>\n\n
\n
History of Apache Kafka<\/h4>\n
Need of Apache Kafka Cluster<\/h4>\n
\n
Audience for Kafka Tutorial<\/h4>\n
Prerequisites to Kafka<\/h4>\n
Kafka Architecture<\/h3>\n
<\/a><\/p>\nKafka Components<\/h3>\n
1. Kafka Topic<\/h3>\n
<\/a><\/p>\n2. Kafka Producer<\/h3>\n
3.\u00a0Kafka Consumer<\/h3>\n
4.\u00a0Kafka Broker<\/h3>\n
5.\u00a0Kafka Zookeeper<\/h3>\n
Log Anatomy<\/h4>\n
<\/a><\/p>\nData Log<\/span><\/h4>\n
Partition in Kafka<\/span><\/h4>\n
<\/a><\/p>\nImportance of Java in Apache Kafka<\/span><\/h4>\n
Kafka Use Cases<\/span><\/h3>\n
Comparisons in Kafka<\/span><\/h3>\n
1. Apache Kafka vs Apache Flume<\/h4>\n
<\/a><\/p>\n
\n<\/b><\/p>\n
\n<\/b><\/p>\n
\n<\/b><\/p>\n
\n<\/b><\/p>\n2. RabbitMQ vs Apache Kafka<\/h4>\n