English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Training wide and deep neural networks (DNNs) require large amounts ofstorage resources such as memory because the intermediate activation data mustbe saved in the memory during forward propagation and then restored forbackward propagation. However, state-of-the-art accelerators such as GPUs areonly equipped with very limited memory capacities due to hardware designconstraints, which significantly limits the maximum batch size and henceperformance speedup when training large-scale DNNs. Traditional memory savingtechniques either suffer from performance overhead or are constrained bylimited interconnect bandwidth or specific interconnect technology. In thispaper, we propose a novel memory-efficient CNN training framework (calledCOMET) that leverages error-bounded lossy compression to significantly reducethe memory requirement for training, to allow training larger models or toaccelerate training. Different from the state-of-the-art solutions that adoptimage-based lossy compressors (such as JPEG) to compress the activation data,our framework purposely adopts error-bounded lossy compression with a stricterror-controlling mechanism. Specifically, we perform a theoretical analysis onthe compression error propagation from the altered activation data to thegradients, and empirically investigate the impact of altered gradients over thetraining process. Based on these analyses, we optimize the error-bounded lossycompression and propose an adaptive error-bound control scheme for activationdata compression. We evaluate our design against state-of-the-art solutionswith five widely-adopted CNNs and ImageNet dataset. Experiments demonstratethat our proposed framework can significantly reduce the training memoryconsumption by up to 13.5X over the baseline training and 1.8X over anotherstate-of-the-art compression-based framework, respectively, with little or noaccuracy loss.

COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression