English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In High Energy Physics (HEP), experimentalists generate large volumes of datathat, when analyzed, helps us better understand the fundamental particles andtheir interactions. This data is often captured in many files of small size,creating a data management challenge for scientists. In order to betterfacilitate data management, transfer, and analysis on large scale platforms, itis advantageous to aggregate data further into a smaller number of largerfiles. However, this translation process can consume significant time andresources, and if performed incorrectly the resulting aggregated files can beinefficient for highly parallel access during analysis on large scaleplatforms. In this paper, we present our case study on parallel I/O strategiesand HDF5 features for reducing data aggregation time, making effective use ofcompression, and ensuring efficient access to the resulting data duringanalysis at scale. We focus on NOvA detector data in this case study, alarge-scale HEP experiment generating many terabytes of data. The lessonslearned from our case study inform the handling of similar datasets, thusexpanding community knowledge related to this common data management task.

A Case Study on Parallel HDF5 Dataset Concatenation for High Energy Physics Data Analysis