Скачать или смотреть Mainlining Databases : Supporting Fast Transactional Workloads on Apache Arrow

Speaker: Tianyu Li, PhD Student at Massachusetts Institute of Technology

The proliferation of modern data processing tools has given rise to open-source columnar data formats like Apache Arrow. These formats help organizations avoid repeated conversion of data to a new format for each application. However, these formats are read-only, and organizations must use a heavy-weight transformation process to load data from on-line transactional processing (OLTP) systems. As a result, DBMSs often fail to take advantage of full network bandwidth when transferring data.

He and his group aim to reduce or even eliminate this overhead by developing a storage architecture for in-memory database management systems (DBMSs) that is aware of the eventual usage of its data and emits columnar storage blocks natively in Apache Arrow. They introduce relaxations to Apache Arrow to efficiently update records and rely on a lightweight transformation process to convert blocks to a read-optimized layout when they are cold. They also describe how to access data from third-party analytical tools with minimal serialization overhead. We implemented our Arrow transactional storage engine and integrated it into the NoisePage DBMS to evaluate our work. These experiments show that the approach achieves comparable performance with dedicated OLTP DBMSs while enabling orders-of-magnitude faster data exports to external data science and machine learning tools than existing methods.