Скачать или смотреть Optimizing data lake infrastructure for sub second query latency

Session: Optimizing data lake infrastructure for sub-second query latency

Emil Emilov, Principal Software Engineer @ Conductor

Emil will share his journey of building and optimizing a data lake infrastructure using various open-source projects and a cloud-native data platform for high-performance user-facing analytics.

The talk will include real-world challenges and solutions around partitioning, custom bucketing, and optimizing query engines to handle massive datasets while achieving sub-second query latency. Attendees will gain insights into the nuances of data skipping and pruning, and best practices for data modeling to avoid large scans.

This session is ideal for developers and data engineers tackling the complexity of scaling analytics on large data lakes, particularly those focused on delivering high-performance, user-facing applications.