English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Memory efficiency is crucial in training deep learning networks onresource-restricted devices. During backpropagation, forward tensors are usedto calculate gradients. Despite the option of keeping those dependencies inmemory until they are reused in backpropagation, some forward tensors can bediscarded and recomputed later from saved tensors, so-called checkpoints. Thisallows, in particular, for resource-constrained heterogeneous environments tomake use of all available compute devices. Unfortunately, the definition ofthese checkpoints is a non-trivial problem and poses a challenge to theprogrammer - improper or excessive recomputations negate the benefit ofcheckpointing.  In this article, we present XEngine, an approach that schedules networkoperators to heterogeneous devices in low memory environments by determiningcheckpoints and recomputations of tensors. Our approach selects suitableresources per timestep and operator and optimizes the end-to-end time forneural networks taking the memory limitation of each device into account. Forthis, we formulate a mixed-integer quadratic program (MIQP) to scheduleoperators of deep learning networks on heterogeneous systems. We compare ourMIQP solver XEngine against Checkmate, a mixed-integer linear programming(MILP) approach that solves recomputation on a single device. Our solver findssolutions that are up to 22.5 % faster than the fastest Checkmate schedule inwhich the network is computed exclusively on a single device. We also findvalid schedules for networks making use of both central processing units andgraphics processing units if memory limitations do not allow schedulingexclusively to the graphics processing unit.

XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environments