English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We design learning rate schedules that minimize regret for SGD-based onlinelearning in the presence of a changing data distribution. We fully characterizethe optimal learning rate schedule for online linear regression via a novelanalysis with stochastic differential equations. For general convex lossfunctions, we propose new learning rate schedules that are robust todistribution shift and we give upper and lower bounds for the regret that onlydiffer by constants. For non-convex loss functions, we define a notion ofregret based on the gradient norm of the estimated models and propose alearning schedule that minimizes an upper bound on the total expected regret.Intuitively, one expects changing loss landscapes to require more exploration,and we confirm that optimal learning rate schedules typically increase in thepresence of distribution shift. Finally, we provide experiments forhigh-dimensional regression models and neural networks to illustrate theselearning rate schedules and their cumulative regret.

Learning Rate Schedules in the Presence of Distribution Shift