Actor Workflows: Reliably orchestrating thousands of Flink clusters at Netflix | Replay 2023

At Netflix, we operate over 12000 Apache Flink clusters, processing over 60 PB of data per day. Reliably managing these clusters pose various challenges such as fault tolerance, concurrency control, and consistency between actual and desired infrastructure state.

In this talk, we'll present how we leveraged Temporal to build a reliable and scalable control plane for the Flink platform at Netflix. We've designed our solution using the actor model implemented via long-running Temporal workflows. We'll discuss the benefits and the challenges that we've encountered while building our architecture.

---

Temporal is the simple, scalable, open source way to write and run reliable cloud applications.

Learn more
Blog: https://temporal.io/blog
How Temporal Works: https://temporal.io/how-temporal-works
Community Slack: https://temporal.io/slack

Developer resources
Docs: https://docs.temporal.io
Courses: https://learn.temporal.io/courses
Support forum: https://community.temporal.io