English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Hyperparameter optimization can be formulated as a bilevel optimizationproblem, where the optimal parameters on the training set depend on thehyperparameters. We aim to adapt regularization hyperparameters for neuralnetworks by fitting compact approximations to the best-response function, whichmaps hyperparameters to optimal weights and biases. We show how to constructscalable best-response approximations for neural networks by modeling thebest-response as a single network whose hidden units are gated conditionally onthe regularizer. We justify this approximation by showing the exactbest-response for a shallow linear network with L2-regularized Jacobian can berepresented by a similar gating mechanism. We fit this model using agradient-based hyperparameter optimization algorithm which alternates betweenapproximating the best-response around the current hyperparameters andoptimizing the hyperparameters using the approximate best-response function.Unlike other gradient-based approaches, we do not require differentiating thetraining loss with respect to the hyperparameters, allowing us to tune discretehyperparameters, data augmentation hyperparameters, and dropout probabilities.Because the hyperparameters are adapted online, our approach discovershyperparameter schedules that can outperform fixed hyperparameter values.Empirically, our approach outperforms competing hyperparameter optimizationmethods on large-scale deep learning problems. We call our networks, whichupdate their own hyperparameters online during training, Self-Tuning Networks(STNs).

Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions