Restate: Stream Processing, but for Microservices

Developing applications in a microservice architecture (MSA) is hard. For all the benefits that MSAs bring over monolithic application architectures, they expose developers to all sorts of tough distributed systems problems, making it non-trivial to build applications that are consistent, scalable, and resilient.

Building applications in an event-driven way is a great start to get MSA complexity under control, by leveraging the guarantees of logs/message queues (like Kafka) to offload some of the problems. Stream processing systems (KStreams, Flink, etc.) can be powerful tools as well. Their tight integration of state, messaging, and application logic yields strong guarantees, like exactly-once state & event-delivery semantics. However, having been built originally for data processing, those systems cannot easily express business logic with complex control-flow, orchestration tasks, or workflows. Existing systems also have very different operational characteristics compared to stateless service deployments.

In this session, we share ideas from a novel system we are developing, called 'Restate'. Our work is inspired by event-sourcing and stream processing systems, but rethought from the ground up for microservices. Restate connects services with each other (and between services and Kafka), and handles crucial aspects like state, communication, failover, and consistency. It does that with a programming model that is crafted not for data processing or analytics, but for services and applications with complex interactions and control flow. The result is an elegant experience of writing applications that are simple, consistent, yet highly resilient and scalable.

We believe that Restate complements the current landscape of event-processing systems, by offering a solution for cases where events facilitate control flow and coordination, rather than real-time data processing.

ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.

#confluent #apachekafka #kafka