English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Understanding the loss surface of neural networks is essential for the designof models with predictable performance and their success in applications.Experimental results suggest that sufficiently deep and wide neural networksare not negatively impacted by suboptimal local minima. Despite recentprogress, the reason for this outcome is not fully understood. Could deepnetworks have very few, if at all, suboptimal local optima? or could all ofthem be equally good? We provide a construction to show that suboptimal localminima (i.e., non-global ones), even though degenerate, exist for fullyconnected neural networks with sigmoid activation functions. The local minimaobtained by our construction belong to a connected set of local solutions thatcan be escaped from via a non-increasing path on the loss curve. For extremelywide neural networks of decreasing width after the wide layer, we prove thatevery suboptimal local minimum belongs to such a connected set. This provides apartial explanation for the successful application of deep neural networks. Inaddition, we also characterize under what conditions the same constructionleads to saddle points instead of local minima for deep neural networks.

Non-attracting Regions of Local Minima in Deep and Wide Neural Networks