English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Deep learning has achieved tremendous success by training increasingly largemodels, which are then compressed for practical deployment. We propose adrastically different approach to compact and optimal deep learning: Wedecouple the Degrees of freedom (DoF) and the actual number of parameters of amodel, optimize a small DoF with predefined random linear constraints for alarge model of arbitrary architecture, in one-stage end-to-end learning.Specifically, we create a recurrent parameter generator (RPG), which repeatedlyfetches parameters from a ring and unpacks them onto a large model with randompermutation and sign flipping to promote parameter decorrelation. We show thatgradient descent can automatically find the best model under constraints withfaster convergence. Our extensive experimentation reveals a log-linearrelationship between model DoF and accuracy. Our RPG demonstrates remarkableDoF reduction and can be further pruned and quantized for additional run-timeperformance gain. For example, in terms of top-1 accuracy on ImageNet, RPGachieves $96\%$ of ResNet18's performance with only $18\%$ DoF (the equivalentof one convolutional layer) and $52\%$ of ResNet34's performance with only$0.25\%$ DoF! Our work shows a significant potential of constrained neuraloptimization in compact and optimal deep learning.

Compact and Optimal Deep Learning with Recurrent Parameter Generators