English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Mini-batch sub-sampling in neural network training is unavoidable, due togrowing data demands, memory-limited computational resources such as graphicalprocessing units (GPUs), and the dynamics of on-line learning. In this study wespecifically distinguish between static mini-batch sub-sampled loss functions,where mini-batches are intermittently fixed during training, resulting insmooth but biased loss functions; and the dynamic sub-sampling equivalent,where new mini-batches are sampled at every loss evaluation, trading bias forvariance in sampling induced discontinuities. These render automatedoptimization strategies such as minimization line searches ineffective, sincecritical points may not exist and function minimizers find spurious,discontinuity induced minima.  This paper suggests recasting the optimization problem to find stochasticnon-negative associated gradient projection points (SNN-GPPs). We demonstratethat the SNN-GPP optimality criterion is less susceptible to sub-samplinginduced discontinuities than critical points or minimizers. We conduct a visualinvestigation, comparing local minimum and SNN-GPP optimality criteria in theloss functions of a simple neural network training problem for a variety ofpopular activation functions. Since SNN-GPPs better approximate the location oftrue optima, particularly when using smooth activation functions with highcurvature characteristics, we postulate that line searches locating SNN-GPPscan contribute significantly to automating neural network training

Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide