From 🐛 to 🦋: Data Pipelines Evolution from Batch to Streaming

Despite data streaming being in most companies’ agenda, transitioning out of consolidated batch systems is not as simple as flipping a switch: new technologies, processes and coding frameworks need to be assessed and then adopted which can make the evolution a long and painful process. But, what if we could keep the same framework? This session explores how Apache Flink can narrow the gap between batch and streaming by keeping the same data pipelines definition while the underlying technology evolves.

We’ll start the journey with a typical batch system, based on a relational database, and then showcase how to evolve it to streaming using Apache Flink and Apache Kafka with minimal changes on the data pipeline definition. We’ll cover query based connectors, mimicking the batch behavior, and then move to more advanced change data capture solutions with Debezium. Finally we’ll touch on critical topics like data validation and late arrival of events and expose strategies on how to minimize related risks.

If you’re thinking about migrating from batch to streaming, but are afraid of any disruption the process may cause in your organization, this session is for you!

ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.

#confluent #apachekafka #kafka