English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

As deep learning models continue to increase in size, the memory requirementsfor training have surged. While high-level techniques like offloading,recomputation, and compression can alleviate memory pressure, they alsointroduce overheads. However, a memory-efficient execution plan that includes areasonable operator execution order and tensor memory layout can significantlyincrease the models' memory efficiency and reduce overheads from high-leveltechniques. In this paper, we propose ROAM which operates on computation graphlevel to derive memory-efficient execution plan with optimized operator orderand tensor memory layout for models. We first propose sophisticated theoriesthat carefully consider model structure and training memory load to supportoptimization for large complex graphs that have not been well supported in thepast. An efficient tree-based algorithm is further proposed to search taskdivisions automatically, along with delivering high performance andeffectiveness to solve the problem. Experiments show that ROAM achieves asubstantial memory reduction of 35.7%, 13.3%, and 27.2% compared to Pytorch andtwo state-of-the-art methods and offers a remarkable 53.7x speedup. Theevaluation conducted on the expansive GPT2-XL further validates ROAM'sscalability.

ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout