Скачать или смотреть CI/CD For Data – Building Data Development Environments With Open Source Stacks

Big Data LDN 2022, 22 Sept, 3:00pm, Data Engineering Theatre
Speaker:
Adi Polak, lakeFS

Building a high quality software product is a breeze compared to building a high quality data product. Because software development being an evolved field has defined development best practices and the tools to implement these practices throughout the software lifecycle. On the contrary, most of the data teams have difficulty incorporating best practices like testing changes in isolation, reproducing a data error or rolling back in case of quality issues in production. Most attempts at doing this fall on one of two extremes--either executed with overly simplified hardcoded sample data that let through errors that will appear with production data. Or, executed in a maintenance-intensive dev environment that requires duplicating the production data, which also ends up massively increasing the risk of a breach or data privacy violation. With an ever-increasing number of organizations relying on data to make business decisions, the cost of data error in production is significantly higher too. Thus it becomes imperative for data teams to ensure the accuracy, resiliency and observability of data in production. So how does one go about extending software development best practices to data products? In this talk, you will learn about industry best practices for data lifecycle management and the open source tools (such as lakeFS) that help implement them.