English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The quadratic complexity of self-attention in Transformers has hindered theprocessing of long text. To alleviate this problem, previous works haveproposed to sparsify the attention matrix, taking advantage of the observationthat crucial information about a token can be derived from its neighbors. Thesemethods typically combine one or another form of local attention and globalattention. Such combinations introduce abrupt changes in contextual granularitywhen going from local to global, which may be undesirable. We believe that asmoother transition could potentially enhance model's ability to capturelong-context dependencies. In this study, we introduce Fovea Transformer, along-context focused transformer that addresses the challenges of capturingglobal dependencies while maintaining computational efficiency. To achievethis, we construct a multi-scale tree from the input sequence, and userepresentations of context tokens with a progressively coarser granularity inthe tree, as their distance to the query token increases. We evaluate our modelon three long-context summarization tasks\footnote{Our code is publiclyavailable at: \textit{https://github.com/ZiweiHe/Fovea-Transformer}}. Itachieves state-of-the-art performance on two of them, and competitive resultson the third with mixed improvement and setback of the evaluation metrics.

Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention