Скачать или смотреть Effective Data Lineage Strategies for Real-Time Systems (Mark Soule, Improving) | Current 24

More: https://current.confluent.io | Data in motion is data on a journey, and understanding that journey is a crucial part of streaming governance. Data lineage enables organizations to unravel the intricacies of real-time data flows, enhance transparency, and build trust across an event-driven landscape. Data lands on a topic, consumed and transformed, then produced to another topic, repeat. In this way, data travels through any number of topics and intermediary systems in real-time before reaching its final destination. This poses a significant hurdle; without detailed message-level tracking, chain of custody and transformations done on that data can be lost.

This session will examine various methodologies for incorporating data lineage into event-driven systems. It will detail the design of an opinionated, streaming-centric solution, applicable across diverse architectures. This solution provides the system with message-level tracking, reconciliation, and replay capability without introducing third-party tools. Next, it will consider how to integrate data lineage capabilities into distributed tracing as it is already widely adopted by many organizations. Finally, it will analyze off-the-shelf solutions and assess their effectiveness in satisfying the requirements of event-driven architectures.

Attendees will acquire a deep understanding of the challenges associated with data lineage, gaining clarity around its complex and fractured landscape. Architects will be equipped with streaming-centric strategies for building data lineage, complete with powerful message-level capabilities. This empowers them to navigate the sea of third-party tools, build a bespoke solution, or build on the back of distributed tracing.

LEARN MORE
► Confluent Developer: https://developer.confluent.io

CONNECT
Subscribe on YouTube:    / @confluentdeveloper  
Community Slack: confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin:   / confluent  
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io

ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.

#apachekafka #apacheflink #confluent