English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Modern deep neural networks (DNNs) often require high memory consumption andlarge computational loads. In order to deploy DNN algorithms efficiently onedge or mobile devices, a series of DNN compression algorithms have beenexplored, including factorization methods. Factorization methods approximatethe weight matrix of a DNN layer with the multiplication of two or multiplelow-rank matrices. However, it is hard to measure the ranks of DNN layersduring the training process. Previous works mainly induce low-rank throughimplicit approximations or via costly singular value decomposition (SVD)process on every training step. The former approach usually induces a highaccuracy loss while the latter has a low efficiency. In this work, we proposeSVD training, the first method to explicitly achieve low-rank DNNs duringtraining without applying SVD on every step. SVD training first decomposes eachlayer into the form of its full-rank SVD, then performs training directly onthe decomposed weights. We add orthogonality regularization to the singularvectors, which ensure the valid form of SVD and avoid gradientvanishing/exploding. Low-rank is encouraged by applying sparsity-inducingregularizers on the singular values of each layer. Singular value pruning isapplied at the end to explicitly reach a low-rank model. We empirically showthat SVD training can significantly reduce the rank of DNN layers and achievehigher reduction on computation load under the same accuracy, comparing to notonly previous factorization methods but also state-of-the-art filter pruningmethods.

Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification