English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In theoretical ML, the teacher-student paradigm is often employed as aneffective metaphor for real-life tuition. The above scheme proves particularlyrelevant when the student network is overparameterized as compared to theteacher network. Under these operating conditions, it is tempting to speculatethat the student ability to handle the given task could be eventually stored ina sub-portion of the whole network. This latter should be to some extentreminiscent of the frozen teacher structure, according to suitable metrics,while being approximately invariant across different architectures of thestudent candidate network. Unfortunately, state-of-the-art conventionallearning techniques could not help in identifying the existence of such aninvariant subnetwork, due to the inherent degree of non-convexity thatcharacterizes the examined problem. In this work, we take a leap forward byproposing a radically different optimization scheme which builds on a spectralrepresentation of the linear transfer of information between layers. Thegradient is hence calculated with respect to both eigenvalues and eigenvectorswith negligible increase in terms of computational and complexity load, ascompared to standard training algorithms. Working in this framework, we couldisolate a stable student substructure, that mirrors the true complexity of theteacher in terms of computing neurons, path distribution and topologicalattributes. When pruning unimportant nodes of the trained student, as follows aranking that reflects the optimized eigenvalues, no degradation in the recordedperformance is seen above a threshold that corresponds to the effective teachersize. The observed behavior can be pictured as a genuine second-order phasetransition that bears universality traits.

How a student becomes a teacher: learning and forgetting through Spectral methods