Скачать или смотреть [GPCE24] Specializing Data Access in a Distributed File System (Generative Pearl)

Specializing Data Access in a Distributed File System (Generative Pearl) (Video, GPCE 2024)
Pratyush Das, Anxhelo Xhebraj, and Tiark Rompf
(Purdue University, USA; Purdue University, USA; Purdue University, USA)

Abstract: We propose DDLoader, a system that embeds information about data

partitioning and data distribution in distributed file systems using a metaprogramming framework. We

demonstrate that this technique has practical benefits for building

applications that interact with distributed file system applications.

By using metaprogramming, we bring the traditional benefits of

staging, such as partial evaluation to the domain of

distributed file system access.

Furthermore, this approach also fuses the

data access code with the computation code, allowing end-to-end optimization.

We test our framework on a real-world use case and show that this approach

allows users to generate code that performs faster than traditional data

loading by up to 12.56x on a single thread and even more when parallelized.

Article: https://doi.org/10.1145/3689484.3690736

ORCID: https://orcid.org/0000-0001-8140-0097, https://orcid.org/0009-0008-6670-6408, https://orcid.org/0000-0002-2068-3238

Video Tags: Staging, Generative Programming, Distributed File Systems, splashws24gpcemain-p70-p, doi:10.1145/3689484.3690736, orcid:0000-0001-8140-0097, orcid:0009-0008-6670-6408, orcid:0000-0002-2068-3238

Presentation at the GPCE 2024 conference, October 21–22, 2024, https://2024.splashcon.org/home/gpce-...
Sponsored by ACM SIGPLAN,