English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large pretrained models are increasingly crucial in modern computer visiontasks. These models are typically used in downstream tasks by end-to-endfinetuning, which is highly memory-intensive for tasks with high-resolutiondata, e.g., video understanding, small object detection, and point cloudanalysis. In this paper, we propose Dynamic Reversible Dual-Residual Networks,or Dr$^2$Net, a novel family of network architectures that acts as a surrogatenetwork to finetune a pretrained model with substantially reduced memoryconsumption. Dr$^2$Net contains two types of residual connections, onemaintaining the residual structure in the pretrained models, and the othermaking the network reversible. Due to its reversibility, intermediateactivations, which can be reconstructed from output, are cleared from memoryduring training. We use two coefficients on either type of residual connectionsrespectively, and introduce a dynamic training strategy that seamlesslytransitions the pretrained model to a reversible network with much highernumerical precision. We evaluate Dr$^2$Net on various pretrained models andvarious tasks, and show that it can reach comparable performance toconventional finetuning but with significantly less memory usage.

Dr$^2$Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning