Скачать или смотреть Learnings from Running Large-scale Apache Iceberg™ Table Management Service

#icebergSummit 2025 breakout session delivered by Anurag Mantripragada and Karuppayya Rajendran from #Apple.

Session Description:
Keeping #ApacheIceberg data and metadata files in check is essential for optimal performance of Iceberg tables. Building a service for maintaining Iceberg table’s health relieves users of the burden of monitoring and intervening when files grow out of hand impacting Iceberg table query performance.

This presentation delves into the intricacies of operating a large-scale service focused on Iceberg table maintenance, practical lessons learnt, and tradeoffs. We will explore other aspects of this service including structuring of the jobs, their resource utilization, performance, and scalability and user experience designed to provide intuitive control and visibility over the table maintenance process. We will also discuss our ongoing efforts to implement autonomous maintenance.

By the end of this session, you'll learn how to build a table management system at scale, and understand the architectural challenges and trade-offs that come with it.