English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

As one of standard approaches to train deep neural networks, dropout has beenapplied to regularize large models to avoid overfitting, and the improvement inperformance by dropout has been explained as avoiding co-adaptation betweennodes. However, when correlations between nodes are compared after training thenetworks with or without dropout, one question arises if co-adaptationavoidance explains the dropout effect completely. In this paper, we propose anadditional explanation of why dropout works and propose a new technique todesign better activation functions. First, we show that dropout can beexplained as an optimization technique to push the input towards the saturationarea of nonlinear activation function by accelerating gradient informationflowing even in the saturation area in backpropagation. Based on thisexplanation, we propose a new technique for activation functions, {\em gradientacceleration in activation function (GAAF)}, that accelerates gradients to floweven in the saturation area. Then, input to the activation function can climbonto the saturation area which makes the network more robust because the modelconverges on a flat region. Experiment results support our explanation ofdropout and confirm that the proposed GAAF technique improves imageclassification performance with expected properties.

Understanding Dropout as an Optimization Trick