Stephan Ewen – Fixin the Hard Bits of Event Processing with Restate & Kafka

Event-driven applications are still hard in 2024, despite all their strong and desirable properties (scalability, resilience, ability to replay, ...). While streaming analytics are supported by a wave of powerful tools (like Flink, KStreams, Materialize, RisingWave, ...), the transactional side of event-driven applications is not quite as far.

Complex per-event control flow, complicated state semantics, and interactions with many external dependencies (APIs/services/...) make those applications challenging. Observability and debuggability are still ways behind what RPC-based architectures offer with today's tools.

In this talk we present 'Restate' (https://restate.dev/), a new open source tool we have been working on. Restate complements systems like Kafka when building event-driven workflows, async tasks, distributed signaling, or even full microservices architectures. While Restate internally is an event-driven system (sharing some similarities with a Kafka/Flink/KStreams crossover), it has a very simple programming model: Durable Execution (for event-handlers and RPC), Durable Promises (for signaling), and Virtual Objects. Those three constructs are the resilient encapsulations of the core ingredients of every distributed application: compute, communication, and state.

In the talk, we will first we'll look practically at common challenging patterns: Where do those tend to break current architectures? How can we address those issues with Kafka/Restate?
burst-y loads / rapid scaling
head-of-the-line waiting on long event processing times
combining Kafka events with webhooks and other signals
Transactionally updating databases
high-fan-out routing to consumer APIs
state machines with long-running transitions
delaying/postponing certain events

Second, we take a look at similarities, synergies, and differences between Restate, Kafka, and Stream-Processing. We look at the way applications are expressed, how the systems organize event logs and event processing, and how this mirrors the differences between analytical- and transactional applications respectively.

Speaker: Stephan Ewen

More: https://2024.berlinbuzzwords.de/sessi...

###

Follow us on Social Media and join the Community!
Mastodon: https://floss.social/@berlinbuzzwords
LinkedIn:   / berlin-buzzwords  
Instagram:   / berlinbuzzwords  

Website: https://2024.berlinbuzzwords.de

Berlin Buzzwords is an event by Plain Schwarz – https://plainschwarz.com