English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A remarkable characteristic of overparameterized deep neural networks (DNNs)is that their accuracy does not degrade when the network's width is increased.Recent evidence suggests that developing compressible representations is keyfor adjusting the complexity of large networks to the learning task at hand.However, these compressible representations are poorly understood. A promisingstrand of research inspired from biology is understanding representations atthe unit level as it offers a more granular and intuitive interpretation of theneural mechanisms. In order to better understand what facilitates increases inwidth without decreases in accuracy, we ask: Are there mechanisms at the unitlevel by which networks control their effective complexity as their width isincreased? If so, how do these depend on the architecture, dataset, andtraining parameters? We identify two distinct types of "frivolous" units thatproliferate when the network's width is increased: prunable units which can bedropped out of the network without significant change to the output andredundant units whose activities can be expressed as a linear combination ofothers. These units imply complexity constraints as the function the networkrepresents could be expressed by a network without them. We also identify howthe development of these units can be influenced by architecture and a numberof training factors. Together, these results help to explain why the accuracyof DNNs does not degrade when width is increased and highlight the importanceof frivolous units toward understanding implicit regularization in DNNs.

Frivolous Units: Wider Networks Are Not Really That Wide