English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural Architecture Search (NAS) has fostered the automatic discovery ofstate-of-the-art neural architectures. Despite the progress achieved with NAS,so far there is little attention to theoretical guarantees on NAS. In thiswork, we study the generalization properties of NAS under a unifying frameworkenabling (deep) layer skip connection search and activation function search. Tothis end, we derive the lower (and upper) bounds of the minimum eigenvalue ofthe Neural Tangent Kernel (NTK) under the (in)finite-width regime using acertain search space including mixed activation functions, fully connected, andresidual neural networks. We use the minimum eigenvalue to establishgeneralization error bounds of NAS in the stochastic gradient descent training.Importantly, we theoretically and experimentally show how the derived resultscan guide NAS to select the top-performing architectures, even in the casewithout training, leading to a train-free algorithm based on our theory.Accordingly, our numerical validation shed light on the design ofcomputationally efficient methods for NAS. Our analysis is non-trivial due tothe coupling of various architectures and activation functions under theunifying framework and has its own interest in providing the lower bound of theminimum eigenvalue of NTK in deep learning theory.

Generalization Properties of NAS under Activation and Skip Connection Search