English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Nondeterminism in neural network optimization produces uncertainty inperformance, making small improvements difficult to discern from run-to-runvariability. While uncertainty can be reduced by training multiple modelcopies, doing so is time-consuming, costly, and harms reproducibility. In thiswork, we establish an experimental protocol for understanding the effect ofoptimization nondeterminism on model diversity, allowing us to isolate theeffects of a variety of sources of nondeterminism. Surprisingly, we find thatall sources of nondeterminism have similar effects on measures of modeldiversity. To explain this intriguing fact, we identify the instability ofmodel training, taken as an end-to-end procedure, as the key determinant. Weshow that even one-bit changes in initial parameters result in modelsconverging to vastly different values. Last, we propose two approaches forreducing the effects of instability on run-to-run variability.

Nondeterminism and Instability in Neural Network Optimization