English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

During typical gradient-based training of deep neural networks, all of themodel's parameters are updated at each iteration. Recent work has shown that itis possible to update only a small subset of the model's parameters duringtraining, which can alleviate storage and communication requirements. In thispaper, we show that it is possible to induce a fixed sparse mask on the model'sparameters that selects a subset to update over many iterations. Our methodconstructs the mask out of the $k$ parameters with the largest Fisherinformation as a simple approximation as to which parameters are most importantfor the task at hand. In experiments on parameter-efficient transfer learningand distributed training, we show that our approach matches or exceeds theperformance of other methods for training with sparse updates while being moreefficient in terms of memory usage and communication costs. We release our codepublicly to promote further applications of our approach.

Training Neural Networks with Fixed Sparse Masks