English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper we explore acceleration techniques for large scale nonconvexoptimization problems with special focuses on deep neural networks. Theextrapolation scheme is a classical approach for accelerating stochasticgradient descent for convex optimization, but it does not work well fornonconvex optimization typically. Alternatively, we propose an interpolationscheme to accelerate nonconvex optimization and call the method Interpolatron.We explain motivation behind Interpolatron and conduct a thorough empiricalanalysis. Empirical results on DNNs of great depths (e.g., 98-layer ResNet and200-layer ResNet) on CIFAR-10 and ImageNet show that Interpolatron can convergemuch faster than the state-of-the-art methods such as the SGD with momentum andAdam. Furthermore, Anderson's acceleration, in which mixing coefficients arecomputed by least-squares estimation, can also be used to improve theperformance. Both Interpolatron and Anderson's acceleration are easy toimplement and tune. We also show that Interpolatron has linear convergence rateunder certain regularity assumptions.

Interpolatron: Interpolation or Extrapolation Schemes to Accelerate Optimization for Deep Neural Networks