Kafka Monitoring on Kubernetes with Datadog - Demo-3

You have deployed this new environment, but you realize maybe—it would be a good idea to monitor the performance, health and capacity of the Kafka cluster.

This article will discuss considerations to keep in mind in your observability journey.

Requirements

Our customer had the following requirements: that this Kafka cluster be hosted on their Kubernetes cluster and monitored using Datadog. Other monitoring environments can be adapted from this experience.

Kafka

There are many Kubernetes-ready implementations of Kafka available from Apache, Confluent and, in this case, Bitnami [https://bitnami.com/stack/kafka/helm]. This is a packaged version that is fairly simple to get up and running on your Kubernetes cluster:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm install my-kafka bitnami/kafka

Side note: most of the time, when you see Kafka, you will also see ZooKeeper, which is used for cluster management. Newer versions of Kafka can run standalone using the Kraft protocol without ZooKeeper. Bitnami provides ZooKeeper as an integrated or separate deployment in the Helm chart, depending on your requirements, and it can have monitoring annotations as well. 

There are a number of options which can be added to a “values.yaml” file based on your requirements. For the purposes of this article, we will only be discussing the metrics portion a bit later on.

Default “values.yaml”: https://github.com/bitnami/charts/blob/main/bitnami/kafka/values.yaml

JMX Metrics

Since Kafka was written in Scala, it runs in a Java virtual machine and can utilize JMX (Java Management Extensions) for its metrics. These metrics can be viewed locally (on your desktop machine) using a Java program called Jconsole