English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Unstructured pruning reduces the memory footprint in deep neural networks(DNNs). Recently, researchers proposed different types of structural pruningintending to reduce also the computation complexity. In this work, we firstsuggest a new measure called mask-diversity which correlates with the expectedaccuracy of the different types of structural pruning. We focus on the recentlysuggested N:M fine-grained block sparsity mask, in which for each block of Mweights, we have at least N zeros. While N:M fine-grained block sparsity allowsacceleration in actual modern hardware, it can be used only to accelerate theinference phase. In order to allow for similar accelerations in the trainingphase, we suggest a novel transposable fine-grained sparsity mask, where thesame mask can be used for both forward and backward passes. Our transposablemask guarantees that both the weight matrix and its transpose follow the samesparsity pattern; thus, the matrix multiplication required for passing theerror backward can also be accelerated. We formulate the problem of finding theoptimal transposable-mask as a minimum-cost flow problem. Additionally, tospeed up the minimum-cost flow computation, we also introduce a fastlinear-time approximation that can be used when the masks dynamically changeduring training. Our experiments suggest a 2x speed-up in the matrixmultiplications with no accuracy degradation over vision and language models.Finally, to solve the problem of switching between different structureconstraints, we suggest a method to convert a pre-trained model withunstructured sparsity to an N:M fine-grained block sparsity model with littleto no training. A reference implementation can be found athttps://github.com/papers-submission/structured_transposable_masks.

Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks