English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In our work, we bridge deep neural network design with numerical differentialequations. We show that many effective networks, such as ResNet, PolyNet,FractalNet and RevNet, can be interpreted as different numericaldiscretizations of differential equations. This finding brings us a brand newperspective on the design of effective deep architectures. We can takeadvantage of the rich knowledge in numerical analysis to guide us in designingnew and potentially more effective deep networks. As an example, we propose alinear multi-step architecture (LM-architecture) which is inspired by thelinear multi-step method solving ordinary differential equations. TheLM-architecture is an effective structure that can be used on any ResNet-likenetworks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. thenetworks obtained by applying the LM-architecture on ResNet and ResNeXtrespectively) can achieve noticeably higher accuracy than ResNet and ResNeXt onboth CIFAR and ImageNet with comparable numbers of trainable parameters. Inparticular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantlycompress ($>50$\%) the original networks while maintaining a similarperformance. This can be explained mathematically using the concept of modifiedequation from numerical analysis. Last but not least, we also establish aconnection between stochastic control and noise injection in the trainingprocess which helps to improve generalization of the networks. Furthermore, byrelating stochastic training strategy with stochastic dynamic system, we caneasily apply stochastic training to the networks with the LM-architecture. Asan example, we introduced stochastic depth to LM-ResNet and achieve significantimprovement over the original LM-ResNet on CIFAR10.

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations