English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We provide a theoretical algorithm for checking local optimality and escapingsaddles at nondifferentiable points of empirical risks of two-layer ReLUnetworks. Our algorithm receives any parameter value and returns: localminimum, second-order stationary point, or a strict descent direction. Thepresence of $M$ data points on the nondifferentiability of the ReLU divides theparameter space into at most $2^M$ regions, which makes analysis difficult. Byexploiting polyhedral geometry, we reduce the total computation down to oneconvex quadratic program (QP) for each hidden node, $O(M)$ (in)equality tests,and one (or a few) nonconvex QP. For the last QP, we show that our specificproblem can be solved efficiently, in spite of nonconvexity. In the benigncase, we solve one equality constrained QP, and we prove that projectedgradient descent solves it exponentially fast. In the bad case, we have tosolve a few more inequality constrained QPs, but we prove that the timecomplexity is exponential only in the number of inequality constraints. Ourexperiments show that either benign case or bad case with very few inequalityconstraints occurs, implying that our algorithm is efficient in most cases.

Efficiently testing local optimality and escaping saddles for ReLU networks