English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this work, we present a new derivative-free optimization method andinvestigate its use for training neural networks. Our method is motivated bythe Ensemble Kalman Filter (EnKF), which has been used successfully for solvingoptimization problems that involve large-scale, highly nonlinear dynamicalsystems. A key benefit of the EnKF method is that it requires only theevaluation of the forward propagation but not its derivatives. Hence, in thecontext of neural networks, it alleviates the need for back propagation andreduces the memory consumption dramatically. However, the method is not a pure"black-box" global optimization heuristic as it efficiently utilizes thestructure of typical learning problems. Promising first results of the EnKF fortraining deep neural networks have been presented recently by Kovachki andStuart. We propose an important modification of the EnKF that enables us toprove convergence of our method to the minimizer of a strongly convex function.Our method also bears similarity with implicit filtering and we demonstrate itspotential for minimizing highly oscillatory functions using a simple example.Further, we provide numerical examples that demonstrate the potential of ourmethod for training deep neural networks.

Never look back - A modified EnKF method and its application to the training of neural networks without back propagation