English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

In artificial neural networks, learning from data is a computationallydemanding task in which a large number of connection weights are iterativelytuned through stochastic-gradient-based heuristic processes over acost-function. It is not well understood how learning occurs in these systems,in particular how they avoid getting trapped in configurations with poorcomputational performance. Here we study the difficult case of networks withdiscrete weights, where the optimization landscape is very rough even forsimple architectures, and provide theoretical and numerical evidence of theexistence of rare - but extremely dense and accessible - regions ofconfigurations in the network weight space. We define a novel measure, which wecall the "robust ensemble" (RE), which suppresses trapping by isolatedconfigurations and amplifies the role of these dense regions. We analyticallycompute the RE in some exactly solvable models, and also provide a generalalgorithmic scheme which is straightforward to implement: define acost-function given by a sum of a finite number of replicas of the originalcost-function, with a constraint centering the replicas around a drivingassignment. To illustrate this, we derive several powerful new algorithms,ranging from Markov Chains to message passing to gradient descent processes,where the algorithms target the robust dense states, resulting in substantialimprovements in performance. The weak dependence on the number of precisionbits of the weights leads us to conjecture that very similar reasoning appliesto more conventional neural networks. Analogous algorithmic schemes can also beapplied to other optimization problems.

Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes