English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recently over-smoothing phenomenon of Transformer-based models is observed inboth vision and language fields. However, no existing work has delved deeper tofurther investigate the main cause of this phenomenon. In this work, we makethe attempt to analyze the over-smoothing problem from the perspective ofgraph, where such problem was first discovered and explored. Intuitively, theself-attention matrix can be seen as a normalized adjacent matrix of acorresponding graph. Based on the above connection, we provide some theoreticalanalysis and find that layer normalization plays a key role in theover-smoothing issue of Transformer-based models. Specifically, if the standarddeviation of layer normalization is sufficiently large, the output ofTransformer stacks will converge to a specific low-rank subspace and result inover-smoothing. To alleviate the over-smoothing problem, we considerhierarchical fusion strategies, which combine the representations fromdifferent layers adaptively to make the output more diverse. Extensiveexperiment results on various data sets illustrate the effect of our fusionmethod.

Revisiting Over-smoothing in BERT from the Perspective of Graph