English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The self-attention mechanism (SAM) is widely used in various fields ofartificial intelligence and has successfully boosted the performance ofdifferent models. However, current explanations of this mechanism are mainlybased on intuitions and experiences, while there still lacks direct modelingfor how the SAM helps performance. To mitigate this issue, in this paper, basedon the dynamical system perspective of the residual neural network, we firstshow that the intrinsic stiffness phenomenon (SP) in the high-precisionsolution of ordinary differential equations (ODEs) also widely exists inhigh-performance neural networks (NN). Thus the ability of NN to measure SP atthe feature level is necessary to obtain high performance and is an importantfactor in the difficulty of training NN. Similar to the adaptive step-sizemethod which is effective in solving stiff ODEs, we show that the SAM is also astiffness-aware step size adaptor that can enhance the model's representationalability to measure intrinsic SP by refining the estimation of stiffnessinformation and generating adaptive attention values, which provides a newunderstanding about why and how the SAM can benefit the model performance. Thisnovel perspective can also explain the lottery ticket hypothesis in SAM, designnew quantitative metrics of representational ability, and inspire a newtheoretic-inspired approach, StepNet. Extensive experiments on several popularbenchmarks demonstrate that StepNet can extract fine-grained stiffnessinformation and measure SP accurately, leading to significant improvements invarious visual tasks.

Understanding Self-attention Mechanism via Dynamical System Perspective