English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Deep learning has become the most powerful machine learning tool in the lastdecade. However, how to efficiently train deep neural networks remains to bethoroughly solved. The widely used minibatch stochastic gradient descent (SGD)still needs to be accelerated. As a promising tool to better understand thelearning dynamic of minibatch SGD, the information bottleneck (IB) theoryclaims that the optimization process consists of an initial fitting phase andthe following compression phase. Based on this principle, we further studytypicality sampling, an efficient data selection method, and propose a newexplanation of how it helps accelerate the training process of the deepnetworks. We show that the fitting phase depicted in the IB theory will beboosted with a high signal-to-noise ratio of gradient approximation if thetypicality sampling is appropriately adopted. Furthermore, this finding alsoimplies that the prior information of the training set is critical to theoptimization process and the better use of the most important data can help theinformation flow through the bottleneck faster. Both theoretical analysis andexperimental results on synthetic and real-world datasets demonstrate ourconclusions.

Drill the Cork of Information Bottleneck by Inputting the Most Important Data