Premium
New Lustre features to improve Lustre metadata and small‐file performance
Author(s) -
Fragalla John,
Loewe Bill,
Kling Petersen Torben
Publication year - 2020
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.5649
Subject(s) - lustre (file system) , metadata , computer science , scalability , namespace , file system , operating system , directory , database , journaling file system , computer file , distributed file system
Summary As HPC I/O evolves beyond the challenges of large I/O performance, several new Lustre features ‐ Distributed Namespace (DNE) 2, Progressive File Layout (PFL), and Data on Metadata (DoM), when combined with flash‐based Lustre targets, can provide metadata and small‐file performance improvements to applications transparently. To improve the execution of single‐directory operations and increase the scalability of overall metadata operations, Striped or Remote Directories with DNE2 can be configured with multiple MDTs. For small‐file I/O, Progressive File Layout with flash and disks can be used to optimize small‐ and large‐I/O by seamlessly storing files on flash‐based MDTs or specific flash OSTs. This paper will share performance results that demonstrate the scalability benefits of metadata performance using striped or remote directories as well as small‐file performance with DoM and Flash OSTs with PFL, both with and without Cray's block I/O accelerator (NXD) configured.