English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Given a training set, a loss function, and a neural network architecture, itis often taken for granted that optimal network parameters exist, and a commonpractice is to apply available optimization algorithms to search for them. Inthis work, we show that the existence of an optimal solution is not alwaysguaranteed, especially in the context of {\em sparse} ReLU neural networks. Inparticular, we first show that optimization problems involving deep networkswith certain sparsity patterns do not always have optimal parameters, and thatoptimization algorithms may then diverge. Via a new topological relationbetween sparse ReLU neural networks and their linear counterparts, we derive --using existing tools from real algebraic geometry -- an algorithm to verifythat a given sparsity pattern suffers from this issue. Then, the existence of aglobal optimum is proved for every concrete optimization problem involving ashallow sparse ReLU neural network of output dimension one. Overall, theanalysis is based on the investigation of two topological properties of thespace of functions implementable as sparse ReLU neural networks: a bestapproximation property, and a closedness property, both in the uniform norm.This is studied both for (finite) domains corresponding to practical trainingon finite training sets, and for more general domains such as the unit cube.This allows us to provide conditions for the guaranteed existence of an optimumgiven a sparsity pattern. The results apply not only to several sparsitypatterns proposed in recent works on network pruning/sparsification, but alsoto classical dense neural networks, including architectures not covered byexisting results.

Does a sparse ReLU network training problem always admit an optimum?