{"id":15178,"date":"2018-05-25T06:05:48","date_gmt":"2018-05-25T06:05:48","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=15178"},"modified":"2018-05-25T06:05:48","modified_gmt":"2018-05-25T06:05:48","slug":"kafka-monitoring","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/kafka-monitoring\/","title":{"rendered":"Kafka Monitoring | Methods & Tools for Kafka Monitoring"},"content":{"rendered":"

In our last Kafka Tutorial<\/strong>, we discussed Kafka Tools<\/strong>. Today, we will see Kafka Monitoring. In this, we will learn the concept of how to Monitor Apache Kafka. <\/span><\/p>\n

Moreover, we will cover all possible\/reasonable Kafka metrics that can help at the time of troubleshooting or Kafka Monitoring. Also, we will discuss audit and Kafka Monitoring tools such as Kafka Monitoring JMX.<\/span><\/p>\n

So, let’s begin with Monitoring in Apache Kafka.<\/p>\n

What is Kafka Monitoring?<\/span><\/h2>\n

As we know Kafka’s cluster<\/strong>-centric design offers strong durability and fault-tolerance. Also, topics are partitioned and replicated across multiple nodes, since Kafka is a distributed system. <\/span><\/p>\n

Moreover, Kafka can be a highly attractive option for data integration, with meaningful performance monitoring and prompt alerting of issues. Basically, when there are troubleshooting Kafka issues application manager collects all performance metrics and alerts to those that require corrective action. <\/span><\/p>\n

Let’s discuss Kafka Monitoring in detail –<\/span><\/p>\n

\"Introduction<\/a>

Introduction to Kafka Monitoring<\/p><\/div>\n

a.\u00a0Track System Resource Utilization<\/h3>\n

To ensure you don’t run out of resources \u201cTrack System Resource Utilization\u201d automatically discover Kafka servers, and also track the resource utilization details like memory, CPU and disk growth over time. <\/span><\/p>\n

However, we ensure that your Apache Kafka server is up and continuously operating as expected. Hence, whenever there are sudden surges in resource consumption or unusual patterns, you can get notified quickly.<\/span><\/p>\n

b.\u00a0Keep Tabs on Threads and JVM usage<\/h3>\n

Kafka relies on Java garbage collection<\/strong> processes to free up memory, as it runs in the Java Virtual Machine (JVM)<\/strong>. The more often the garbage collection will run, the more activity in your Kafka cluster occurs. <\/span><\/p>\n

Make sure that started threads don\u2019t overload the server’s memory, also ensure to track JVM heap sizes. Moreover, to prevent performance bottlenecks in your system, track thread usage with metrics like Daemon, Peak, and Live Thread Count.<\/span><\/p>\n

c.\u00a0Understand Broker, Controller and Replication Statistics<\/h3>\n

In a Kafka cluster, for managing the states of partitions and replicas and for performing administrative tasks like reassigning partitions, one of the brokers serves as the controller, that controller is responsible. <\/span><\/p>\n

In order to see which Kafka broker<\/strong> was the controller when an issue occurred, monitor active controllers and to prevent service interruptions, offline partitions count. <\/span><\/p>\n

Also, it monitors the broker’s Log flush latency – However, the longer it takes to flush log to disk, the more the pipeline breaks up. In addition, to know if replication is going as fast as configured, it tracks under-replicated partitions.<\/span><\/p>\n

d.\u00a0Monitor Network and Topic Details<\/h3>\n

Get the full picture of network usage on your host, track network throughput or aggregate incoming and outgoing byte rate on your broker topics for more information as to where potential bottlenecks lie. Make informed decisions.<\/span><\/p>\n

e.\u00a0Fix performance problems faster<\/h3>\n

If there are performance issues with the components of Apache Kafka, get instant notifications. Also, find out which application is causing the excessive load, and become aware of performance bottlenecks. Before your end users experience issues, take quick remedial actions.<\/span><\/p>\n

Metrics and Alerts in Kafka Monitoring\u00a0<\/span><\/h2>\n

Basically, our general rule of thumb is \u201ccollect all possible\/reasonable metrics that can help when troubleshooting, alert only to those that require an action from you\u201d.<\/span><\/p>\n

a. Metrics on which we do graph and alerting<\/strong><\/p>\n