Kafka - Exactly once semantics with Matthias J. Sax

Hey Everyone,

In this episode I and Matthias did a deep dive into the world of exactly once semantics in Kafka. We discussed how Kafka has implemented EOS and how failures of different components are handled.
This talk will also provide you different patterns Kafka uses for maintaining reliability, durability and correctness in the system.

Enjoy!

Chapters:
00:00 Introduction
03:48 What is Exactly once semantics in Kafka?
07:39 Building blocks of Exactly once semantics
10:20 Why is exactly once a hard problem?
12:35 Idempotent producer - how is it implemented?
24:45 What happens when the producer crashes?
32:45 What is a transaction in Kafka world?
33:30 Transactional producer - how it is implemeted?
48:12 What happens when a producer fails? Abort?
51:19 What happens when a broker dies?
55:28 How is high watermark used?
58:48 EOS in stream processing
01:07:25 Topology changes in stream processing
01:18:00 Use cases of EOS

I hope you liked the discussion and learnt about EOS in kafka. Please don't forget to hit the like button and subscribe to channel. More and more talks coming soon.

Important links:
Matthias:   / mjsax  
Confluent blog: https://www.confluent.io/blog/exactly...
Enabling EOS in Kafka Streams: https://www.confluent.io/blog/enablin...
My Discord Server:   / discord  
Connect with me on Linkedin:   / kaivalya-apte-2217221a  

Join my discord server, where I go LIVE frequently.

Cheers,
The GeekNarrator