Скачать или смотреть Resolving Kafka Offsets Issues in Strimzi Kafka with Structured Streaming

Discover how to fix the Kafka offsets issue in your Spark Structured Streaming job by adjusting retention configuration settings
---
This video is based on the question https://stackoverflow.com/q/73505391/ asked by the user 'Karan Alang' ( https://stackoverflow.com/u/6843103/ ) and on the answer https://stackoverflow.com/a/73547708/ provided by the user 'Karan Alang' ( https://stackoverflow.com/u/6843103/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Strimzi Kafka + Structured Streaming - Offsets not available (is this because of segment.bytes)?

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding the Kafka Offsets Issue in Spark Structured Streaming

If you're running a Spark Structured Streaming job that consumes data from Kafka, you may encounter some frustrating challenges. One common issue arises when your Spark job suddenly stops reading data from Kafka after running for a while. There can be several reasons for such behavior, but one key culprit often lies in the configuration settings of your Kafka topic.

In this guide, we will dive into a specific scenario involving an Apache Spark structured streaming job that encounters Kafka offset issues, and walk through the solution step-by-step.



The Problem

Scenario Overview

Imagine you're utilizing Spark on Google Cloud Dataproc and have a long-running structured streaming job that reads data from Kafka every 10 minutes. Your Kafka topic is configured with 3 partitions and has a retention period set to 3 days. However, after a few hours of operation, your Spark application stops receiving messages from Kafka.

The peculiar solution? Deleting the Google Cloud bucket used for checkpointing and restarting the job brings the stream back to life. This raises questions about the underlying issue in the Kafka configuration, particularly regarding the retention settings and offsets.

Key Error Messages

From the logs, you may notice messages indicating that the consumer is experiencing a poll timeout. Typical entries may include:

poll() was longer than the configured max.poll.interval.ms

Indications that the assigned partitions are getting lost

The consumer trying to re-join the group fails due to a MemberIdRequiredException

This scenario suggests a misalignment between the offsets maintained in the checkpoint location and the available offsets in Kafka.



Solution Breakdown

Step 1: Review Your Kafka Topic Configuration

The first thing you should check is the settings applied when you created your Kafka topic. Let's take a look at the relevant configuration:

[[See Video to Reveal this Text or Code Snippet]]

Problem: The retention.ms value was set too low at 259200 (which represents a mere 259,200 milliseconds or approximately 4 hours) instead of the intended 259200000 milliseconds, which equals 3 days. This setting means that Kafka would delete any messages older than 4 hours, resulting in your consumer being unable to access older offsets.

Step 2: Adjust the Retention Configuration

To resolve this issue:

Update your Kafka topic configuration for retention.ms to ensure it matches the actual retention period you need, such as 259200000 (3 days).

Apply this configuration and recreate the Kafka topic with the updated settings if necessary.

Step 3: Validate Offsets After Changes

Once you've updated the retention settings, monitor the offsets in both your Kafka topic and the checkpoint bucket. Keep an eye on the following:

Offsets in Kafka: Verify that the current and oldest offsets are now properly aligned with the retention policy.

Checkpoint Data: The offsets in the checkpoint should no longer be lower than the oldest offset of the topic.

Step 4: Restart Your Spark Job

After making the necessary changes, restart your Spark job. Watch the logs to ensure that it can now read data continuously without interruptions. If the job resumes correctly and processes the data as expected, you’ve successfully resolved the issue.



Conclusion

Setting up streaming jobs with Kafka and Spark can sometimes lead to perplexing issues related to offsets and retention settings. By thoroughly examining the Kafka topic's configuration, particularly the retention.ms parameter, and ensuring its alignment with your data handling requirements, you can effectively resolve Kafka offset iss