English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Structured matrices, such as those derived from Kronecker products (KP), areeffective at compressing neural networks, but can lead to unacceptable accuracyloss when applied to large models. In this paper, we propose the notion ofdoping -- addition of an extremely sparse matrix to a structured matrix. Dopingfacilitates additional degrees of freedom for a small number of parameters,allowing them to independently diverge from the fixed structure. To train LSTMswith doped structured matrices, we introduce the additional parameter matrixwhile slowly annealing its sparsity level. However, we find that performancedegrades as we slowly sparsify the doping matrix, due to co-matrix adaptation(CMA) between the structured and the sparse matrices. We address this overdependence on the sparse matrix using a co-matrix dropout regularization (CMR)scheme. We provide empirical evidence to show that doping, CMA and CMR areconcepts generally applicable to multiple structured matrices (KroneckerProduct, LMF, Hybrid Matrix Decomposition). Additionally, results with dopedkronecker product matrices demonstrate state-of-the-art accuracy at largecompression factors (10 - 25x) across 4 natural language processingapplications with minor loss in accuracy. Doped KP compression techniqueoutperforms previous state-of-the art compression results by achieving 1.3 -2.4x higher compression factor at a similar accuracy, while also beating strongalternatives like pruning and low-rank methods by a large margin (8% or more).Additionally, we show that doped KP can be deployed on commodity hardware usingthe current software stack and achieve 2.5 - 5.5x inference run-time speed-upover baseline.

Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices