Observability of Streaming Kafka Applications by Tim van Baarsen - Software developer at ING

You build your streaming applications and event-driven microservices using Apache Kafka. Are your systems observable enough without depending only on the broker-side metrics and application logs? Can you track down the root cause during incidents, or do you hope everything will be fine after a restart?

In this talk, Tim & Kosta will take you on their observability journey by sharing pitfalls and knowledge our team gained over the last couple of years.

We will answer questions like:

• Do you understand how to expose and use your client-side Kafka metrics?
• JMX, Metric interceptors, Micrometer where to start?
• Why is there a difference between the values of client-side and broker-side metrics?
• Learn how client-side consumer lag metrics can differ from the lag calculated on the cluster.
• What is the right way to use and interpret them?
• Can you measure latency through your complete stack using distributed tracing? • OpenTelemetry, Jaeger & Zipkin, what to pick?

During a step-by-step demo, we will look into different real-life examples and scenarios to demonstrate how to bring the observability of your Kafka applications to the next level.

About Tim:
Tim van Baarsen is a creative software developer at ING Bank in the Netherlands and has been in the software development business for almost 15 years. He is a strong proponent of open-source technology, a big fan of the Spring Framework since the early versions, and his interests lie in building scalable distributed systems. Tim enjoys speaking about his passion for the Spring ecosystem and Apache Kafka at both internal ING events as well at meetups and conferences.