Скачать или смотреть How to Achieve Transactional Publish to Kafka: A Practical Approach

Discover how to implement transactional message publishing to Kafka without losing messages, while avoiding complex synchronous semantics.
---
This video is based on the question https://stackoverflow.com/q/65080481/ asked by the user 'Jonathan' ( https://stackoverflow.com/u/987457/ ) and on the answer https://stackoverflow.com/a/65491993/ provided by the user 'Jonathan' ( https://stackoverflow.com/u/987457/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Transactional Publish to Kafka

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Navigating the Challenge of Transacting to Kafka

In today’s data-driven world, the ability to send messages reliably from a database to Kafka is essential for many applications. However, ensuring that a message is sent only after a successful database transaction can be tricky. This problem arises when it's necessary to avoid losing messages due to system outages or transaction failures. So, how can one achieve a seamless transactional publish to Kafka?

Understanding the Problem

When you want to send a message to Kafka based on a database transaction, there are several potential pitfalls:

Power Loss: If the server loses power after the transaction is committed but before the message is sent to Kafka, the message will be lost.

Network Failures: Similar to power issues, if the network connection drops while sending a message, it could result in lost data.

Transaction Order: If the message is sent before the database transaction is confirmed, it complicates ensuring that messages are only published if necessary.

Given these challenges, is there a structured way to ensure that no messages are lost regardless of what happens during the transaction process? Let's break down a potential solution.

Proposed Solution Outline

The original concept proposed involves several steps to ensure reliability while keeping track of outgoing messages. Here’s a breakdown of the solution:

Step 1: Create an Event Table

An additional event table will be added to the database. This table acts like a holding area for messages intended for Kafka.

Step 2: Store Events Temporarily

Instead of sending messages directly to Kafka, the application will first log the events in this newly created table. This operation occurs before committing the database transaction, ensuring that the event is safely stored.

Step 3: Implement a Kafka Message Processor

Launch a separate process that reads from the event table and posts messages to Kafka. This process will follow a journal-style approach to handle each message reliably.

Detailed Processing Steps

The Kafka message processor operates with several key steps:

Read Unsent Messages: It starts by querying for unsent messages in the event table.

Mark as In Progress: When a message is picked for processing, it gets marked as in progress in the database. This way, even if it fails later, it's clear that the message was being handled.

Post to Kafka: The message is then sent to Kafka.

Error Handling: If the sending fails:

The process can redo the attempt or mark the message for future processing.

Confirmation: On successful send:

The message is read from Kafka to confirm it has been received.

Finally, it gets marked as sent in the database, which indicates to the application that the message has been successfully processed.

Recovery Mechanism

It’s also essential to implement a recovery mechanism in case of outages:

If there’s a failure reading from Kafka, consider those messages lost and potentially trigger a re-send of messages that remain in the in progress state.

Rethinking the Need for Transactional Guarantees

After all this analysis, it’s crucial to note that achieving exactly-once semantics with Kafka may not always be necessary. In many scenarios, businesses tolerate some level of message loss or accept duplicates. Before implementing complicated solutions, assess your requirements thoroughly:

Can some message loss be acceptable? Often, applications don't need strict transactional guarantees.

Revisiting Your Architecture: If you require strong consistency or confirmation, consider whether a messaging system like Kafka is appropriate for those needs.

Conclusion

Transactional publishing in Kafka can indeed be a challenge, especially with the risks of losing messages during