English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The training of stochastic neural network models with binary ($\pm1$) weightsand activations via continuous surrogate networks is investigated. We derivenew surrogates using a novel derivation based on writing the stochastic neuralnetwork as a Markov chain. This derivation also encompasses existing variantsof the surrogates presented in the literature. Following this, we theoreticallystudy the surrogates at initialisation. We derive, using mean field theory, aset of scalar equations describing how input signals propagate through therandomly initialised networks. The equations reveal whether so-called criticalinitialisations exist for each surrogate network, where the network can betrained to arbitrary depth. Moreover, we predict theoretically and confirmnumerically, that common weight initialisation schemes used in standardcontinuous networks, when applied to the mean values of the stochastic binaryweights, yield poor training performance. This study shows that, contrary tocommon intuition, the means of the stochastic binary weights should beinitialised close to $\pm 1$, for deeper networks to be trainable.

Critical initialisation in continuous approximations of binary neural networks