English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Over the past few years, softmax and SGD have become a commonly usedcomponent and the default training strategy in CNN frameworks, respectively.However, when optimizing CNNs with SGD, the saturation behavior behind softmaxalways gives us an illusion of training well and then is omitted. In thispaper, we first emphasize that the early saturation behavior of softmax willimpede the exploration of SGD, which sometimes is a reason for model convergingat a bad local-minima, then propose Noisy Softmax to mitigating this earlysaturation issue by injecting annealed noise in softmax during each iteration.This operation based on noise injection aims at postponing the early saturationand further bringing continuous gradients propagation so as to significantlyencourage SGD solver to be more exploratory and help to find a betterlocal-minima. This paper empirically verifies the superiority of the earlysoftmax desaturation, and our method indeed improves the generalization abilityof CNN model by regularization. We experimentally find that this earlydesaturation helps optimization in many tasks, yielding state-of-the-art orcompetitive results on several popular benchmark datasets.

Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation