English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The Fisher information matrix (FIM) is fundamental to understanding thetrainability of deep neural nets (DNN), since it describes the parameterspace's local metric. We investigate the spectral distribution of theconditional FIM, which is the FIM given a single sample, by focusing onfully-connected networks achieving dynamical isometry. Then, while dynamicalisometry is known to keep specific backpropagated signals independent of thedepth, we find that the parameter space's local metric linearly depends on thedepth even under the dynamical isometry. More precisely, we reveal that theconditional FIM's spectrum concentrates around the maximum and the value growslinearly as the depth increases. To examine the spectrum, considering randominitialization and the wide limit, we construct an algebraic methodology basedon the free probability theory. As a byproduct, we provide an analysis of thesolvable spectral distribution in two-hidden-layer cases. Lastly, experimentalresults verify that the appropriate learning rate for the online training ofDNNs is in inverse proportional to depth, which is determined by theconditional FIM's spectrum.

The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry