English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Heavy-tailed distributions have been studied in statistics, random matrixtheory, physics, and econometrics as models of correlated systems, among otherdomains. Further, heavy-tail distributed eigenvalues of the covariance matrixof the weight matrices in neural networks have been shown to empiricallycorrelate with test set accuracy in several works (e.g. arXiv:1901.08276), buta formal relationship between heavy-tail distributed parameters andgeneralization bounds was yet to be demonstrated. In this work, the compressionframework of arXiv:1802.05296 is utilized to show that matrices with heavy-taildistributed matrix elements can be compressed, resulting in networks withsparse weight matrices. Since the parameter count has been reduced to a sum ofthe non-zero elements of sparse matrices, the compression framework allows usto bound the generalization gap of the resulting compressed network with anon-vacuous generalization bound. Further, the action of these matrices on avector is discussed, and how they may relate to compression and resilientclassification is analyzed.

Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds