English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large neural networks excel in many domains, but they are expensive to trainand fine-tune. A popular approach to reduce their compute or memoryrequirements is to replace dense weight matrices with structured ones (e.g.,sparse, low-rank, Fourier transform). These methods have not seen widespreadadoption (1) in end-to-end training due to unfavorable efficiency--qualitytradeoffs, and (2) in dense-to-sparse fine-tuning due to lack of tractablealgorithms to approximate a given dense weight matrix. To address these issues,we propose a class of matrices (Monarch) that is hardware-efficient (they areparameterized as products of two block-diagonal matrices for better hardwareutilization) and expressive (they can represent many commonly used transforms).Surprisingly, the problem of approximating a dense weight matrix with a Monarchmatrix, though nonconvex, has an analytical optimal solution. These propertiesof Monarch matrices unlock new ways to train and fine-tune sparse and densemodels. We empirically validate that Monarch can achieve favorableaccuracy-efficiency tradeoffs in several end-to-end sparse trainingapplications: speeding up ViT and GPT-2 training on ImageNet classification andWikitext-103 language modeling by 2x with comparable model quality, andreducing the error on PDE solving and MRI reconstruction tasks by 40%. Insparse-to-dense training, with a simple technique called "reversesparsification," Monarch matrices serve as a useful intermediate representationto speed up GPT-2 pretraining on OpenWebText by 2x without quality drop. Thesame technique brings 23% faster BERT pretraining than even the very optimizedimplementation from Nvidia that set the MLPerf 1.1 record. In dense-to-sparsefine-tuning, as a proof-of-concept, our Monarch approximation algorithm speedsup BERT fine-tuning on GLUE by 1.7x with comparable accuracy.

Monarch: Expressive Structured Matrices for Efficient and Accurate Training