Скачать или смотреть Setting Up a Small Kafka Cluster on Bare Metal: Overcoming Common Issues

Learn how to effectively set up a small Kafka cluster using Kraft on bare metal. This blog covers common pitfalls and their solutions for a smooth learning experience.
---
This video is based on the question https://stackoverflow.com/q/77090922/ asked by the user 'Caio Favero' ( https://stackoverflow.com/u/1954874/ ) and on the answer https://stackoverflow.com/a/77093406/ provided by the user 'OneCricketeer' ( https://stackoverflow.com/u/2308683/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Trying to setup a small Kafka cluster on bare metal using Kraft

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Setting Up a Small Kafka Cluster on Bare Metal: Overcoming Common Issues

Setting up a small Kafka cluster can provide a great opportunity to learn and experiment with streaming data applications. However, the process can sometimes lead to unexpected problems, especially when configuring multiple brokers and controllers. This guide will guide you through a common issue faced while setting up a Kafka cluster using Kraft on bare metal and provide detailed solutions.

Introduction to the Problem

In this scenario, you are attempting to establish a Kafka cluster with one controller and two brokers on a single machine. Despite successfully starting all three components, you encounter warnings when trying to produce and consume messages from a test topic. The key warning indicates that there are "leader brokers without a matching listener."

Configuration Insight

Your current configuration included these components:

1 Controller

2 Brokers

Your Kafka setup utilizes the following roles:

Controller (Node ID: 1)

Broker and Controller (Node ID: 2)

Broker (Node ID: 3)

The server.properties files for each component were crafted carefully. However, there were important oversights in the listener names that caused the warnings upon message production and consumption.

Understanding the Configuration Issue

The main problem lies in the naming conventions of your listeners across the different nodes. Here are the key points to address:

Listener Names Consistency:

All listeners should have consistent names across your controllers and brokers. This means avoiding any numerical suffixes.

For example, instead of naming listeners as CONTROLLER1, CONTROLLER2, and so on, use identical names such as CONTROLLER for all controller listener configurations.

Inter-Broker Listener Configuration:

The inter.broker.listener.name must match for partitions to be replicated correctly. A mismatch can lead to failures in replication and cause errors when trying to access the data.

Controller Listener Name Consistency:

If the controller.listener.names are not consistent between nodes in the quorum voters list, it can confuse the Kraft protocol when attempting leader elections.

Suggested Configuration Changes

Here's a revised structure of your configuration settings across nodes:

Controller Configuration (Node ID: 1)

[[See Video to Reveal this Text or Code Snippet]]

Broker and Controller Configuration (Node ID: 2)

[[See Video to Reveal this Text or Code Snippet]]

Broker Configuration (Node ID: 3)

[[See Video to Reveal this Text or Code Snippet]]

Testing Your Configuration

After making these changes, restart your Kafka components using the command-line interface (CLI) and test producing/consuming messages again. If everything is configured correctly, the warnings should disappear, and you should be able to interact with your test topic seamlessly.

Conclusion

Setting up a Kafka cluster on bare metal can be challenging, especially when navigating listener configurations. By ensuring that all listener names are consistent and correctly configured across the components, you can avoid common pitfalls and smoothly operate your Kafka setup for learning and experimentation.

Make sure to revisit your configurations regularly and adhere to best practices to create a reliable Kafka environment.