Evolving Storage Paradigms in Apache Pulsar: From Tiered Storage to Lakehouse Integration

Pulsar Virtual Summit Europe 2024

In this session, we will explore the transformative journey of storage solutions within Apache Pulsar, highlighting the shift towards efficient data management in response to escalating data volumes and storage costs. Initially designed to decouple compute from storage, Apache Pulsar has provided significant benefits through this architecture, including enhanced scalability and flexibility. However, the surge in data generation necessitated the introduction of Tiered Storage, a pivotal development aimed at curbing storage costs while maintaining performance.

Despite its advantages, Tiered Storage encountered challenges, such as the absence of an open format for data storage and limited integration with emerging data management architectures, notably Lakehouses. Addressing these limitations, we introduced Lakehouse Storage, a sophisticated solution that seamlessly offloads data to popular Lakehouse platforms like Delta Lake, Apache Iceberg, and Apache Hudi. This integration not only reduces the operational overhead but also enriches data accessibility and analysis.
The session will delve into how Lakehouse Storage empowers organizations to leverage diverse analytical tools, including Spark SQL and AWS Athena, for dynamic data querying within their Lakehouse environments. Moreover, we will discuss the future-ready integration of Lakehouse Storage with the Ursa engine—a groundbreaking development in Apache Pulsar's ecosystem, which operates without persistent storage, further optimizing data fluidity and system efficiency.

Join us to uncover how these advancements in Apache Pulsar's storage solutions are setting new benchmarks for data management and analytics in the era of big data.

Presented by Kundan Vyas from StreamNative