English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this work, we propose a novel technique to boost training efficiency of aneural network. Our work is based on an excellent idea that whitening theinputs of neural networks can achieve a fast convergence speed. Given thewell-known fact that independent components must be whitened, we introduce anovel Independent-Component (IC) layer before each weight layer, whose inputswould be made more independent. However, determining independent components isa computationally intensive task. To overcome this challenge, we propose toimplement an IC layer by combining two popular techniques, Batch Normalizationand Dropout, in a new manner that we can rigorously prove that Dropout canquadratically reduce the mutual information and linearly reduce the correlationbetween any pair of neurons with respect to the dropout layer parameter $p$. Asdemonstrated experimentally, the IC layer consistently outperforms the baselineapproaches with more stable training process, faster convergence speed andbetter convergence limit on CIFAR10/100 and ILSVRC2012 datasets. Theimplementation of our IC layer makes us rethink the common practices in thedesign of neural networks. For example, we should not place Batch Normalizationbefore ReLU since the non-negative responses of ReLU will make the weight layerupdated in a suboptimal way, and we can achieve better performance by combiningBatch Normalization and Dropout together as an IC layer.

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks