English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Our goal is compression of massive-scale grid-structured data, such as themulti-terabyte output of a high-fidelity computational simulation. For suchdata sets, we have developed a new software package called TuckerMPI, aparallel C++/MPI software package for compressing distributed data. Theapproach is based on treating the data as a tensor, i.e., a multidimensionalarray, and computing its truncated Tucker decomposition, a higher-orderanalogue to the truncated singular value decomposition of a matrix. The resultis a low-rank approximation of the original tensor-structured data. Compressionefficiency is achieved by detecting latent global structure within the data,which we contrast to most compression methods that are focused on localstructure. In this work, we describe TuckerMPI, our implementation of thetruncated Tucker decomposition, including details of the data distribution andin-memory layouts, the parallel and serial implementations of the key kernels,and analysis of the storage, communication, and computational costs. We testthe software on 4.5 terabyte and 6.7 terabyte data sets distributed across 100sof nodes (1000s of MPI processes), achieving compression rates between100-200,000$\times$ which equates to 99-99.999% compression (depending on thedesired accuracy) in substantially less time than it would take to even readthe same dataset from a parallel filesystem. Moreover, we show that our methodalso allows for reconstruction of partial or down-sampled data on a singlenode, without a parallel computer so long as the reconstructed portion is smallenough to fit on a single machine, e.g., in the instance ofreconstructing/visualizing a single down-sampled time step or computing summarystatistics.

TuckerMPI: A Parallel C++/MPI Software Package for Large-scale Data Compression via the Tucker Tensor Decomposition