English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In spite of several notable efforts, explaining the generalization ofdeterministic non-smooth deep nets, e.g., ReLU-nets, has remained challenging.Existing approaches for deterministic non-smooth deep nets typically need tobound the Lipschitz constant of such deep nets but such bounds are quite large,may even increase with the training set size yielding vacuous generalizationbounds. In this paper, we present a new family of de-randomized PAC-Bayesmargin bounds for deterministic non-convex and non-smooth predictors, e.g.,ReLU-nets. Unlike PAC-Bayes, which applies to Bayesian predictors, thede-randomized bounds apply to deterministic predictors like ReLU-nets. Aspecific instantiation of the bound depends on a trade-off between the(weighted) distance of the trained weights from the initialization and theeffective curvature (`flatness') of the trained predictor.  To get to these bounds, we first develop a de-randomization argument fornon-convex but smooth predictors, e.g., linear deep networks (LDNs), whichconnects the performance of the deterministic predictor with a Bayesianpredictor. We then consider non-smooth predictors which for any given inputrealized as a smooth predictor, e.g., ReLU-nets become some LDNs for any giveninput, but the realized smooth predictors can be different for differentinputs. For such non-smooth predictors, we introduce a new PAC-Bayes analysiswhich takes advantage of the smoothness of the realized predictors, e.g., LDN,for a given input, and avoids dependency on the Lipschitz constant of thenon-smooth predictor. After careful de-randomization, we get a bound for thedeterministic non-smooth predictor. We also establish non-uniform samplecomplexity results based on such bounds. Finally, we present extensiveempirical results of our bounds over changing training set size and randomnessin labels.

De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors