Distributed Affix-Based Metadata Search in Self-Describing Data Files – Wei Zhang, Ph.D, LBNL

From the 2024 HDF5 User Group Meeting (#HUG24) held August 5-7, 2024 in Chicago, IL.

Hermes: A Heterogeneous-Aware Multi-Tiered Distributed I/O Buffering System – Luke Logan, Research Software Engineer at Gnosis Research Center

Modern High-Performance Computing (HPC) systems are adding extra layers to the memory and storage hierarchy, named deep memory and storage hierarchy (DMSH), to increase I/O performance. New hardware technologies, such as NVMe and SSD, have been introduced in burst buffer installations to reduce the pressure for external storage and boost the burstiness of modern I/O systems. DMSH has demonstrated its strength and potential in practice. However, each layer of DMSH is an independent heterogeneous system and data movement among more layers is significantly more complex even without considering heterogeneity. How to efficiently utilize the DMSH is a subject of research facing the HPC community. In this paper, we present the design and implementation of Hermes: a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering system. Hermes enables, manages, supervises, and, in some sense, extends I/O buffering to fully integrate into the DMSH. We introduce three novel data placement policies to efficiently utilize all layers and we present three novel techniques to perform memory, metadata, and communication management in hierarchical buffering systems. Our evaluation shows that, in addition to automatic data movement through the hierarchy, Hermes can significantly accelerate I/O and outperforms by more than 2x state-of-the-art buffering platforms.

For more information on this conference including all sessions and slide decks, visit https://www.hdfgroup.org/hug/hug24/ag... To learn more about upcoming HUG events, please visit https://www.hdfgroup.org/hug