Decoupling Compute and Storage for Stream Processing Systems by Yingjun Wu - CEO RisingWave Labs

DSDSD - THE DUTCH SEMINAR ON DATA SYSTEMS DESIGN:
We hold bi-weekly talks on Fridays from 3:30 PM to 5 PM CET for and by researchers and practitioners designing (and implementing) data systems. The objective is to establish a new forum for the Dutch Data Systems community to come together, foster collaborations between its members, and bring in high-quality international speakers. We would like to invite all researchers, especially also Ph.D. students, who are working on related topics to join the events. It is an excellent opportunity to receive feedback early on from researchers in your field.

Website: https://dsdsd.da.cwi.nl/
Twitter:   / dsdsdnl  

Abstract
=====
Stream processing is an essential part of modern data infrastructure, but building an efficient and scalable stream processing system can be challenging. Decoupling compute and storage architecture has become an effective way to address these challenges.

In this talk, we discuss the benefits and limitations of the decoupled compute and storage architecture in stream processing systems. We find that, while decoupling compute and storage can help achieve infinite scalability, this approach can lead to data consistency and high latency issues, especially when processing complex continuous queries that require managing extra-large internal states. We then present our solution to address the challenges by implementing a tiered storage mechanism. The tiered storage approach utilizes a combination of high-performance and low-cost storage tiers to minimize data movement between the compute and storage layers while maintaining efficient processing. By the the end of the talk, we will present experimental results that demonstrate the balance between performance and cost-efficiency achieved by our proposed approach.


Speaker Biography
=====
Yingjun Wu is the founder of RisingWave Labs (https://www.risingwave.com/), a database company developing RisingWave, a distributed SQL database for stream processing. Before running the company, Yingjun was a software engineer at the Redshift team, Amazon Web Services, and a researcher at the Database group, IBM Almaden Research Center. Yingjun received his PhD degree from National University of Singapore, and was a visiting PhD at Carnegie Mellon University. He has been working in the field of stream processing and database systems for over a decade.