English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We present a generalization of Transformers to any-order permutationinvariant data (sets, graphs, and hypergraphs). We begin by observing thatTransformers generalize DeepSets, or first-order (set-input) permutationinvariant MLPs. Then, based on recently characterized higher-order invariantMLPs, we extend the concept of self-attention to higher orders and proposehigher-order Transformers for order-$k$ data ($k=2$ for graphs and $k>2$ forhypergraphs). Unfortunately, higher-order Transformers turn out to haveprohibitive complexity $\mathcal{O}(n^{2k})$ to the number of input nodes $n$.To address this problem, we present sparse higher-order Transformers that havequadratic complexity to the number of input hyperedges, and further adopt thekernel attention approach to reduce the complexity to linear. In particular, weshow that the sparse second-order Transformers with kernel attention aretheoretically more expressive than message passing operations while having anasymptotically identical complexity. Our models achieve significant performanceimprovement over invariant MLPs and message-passing graph neural networks inlarge-scale graph regression and set-to-(hyper)graph prediction tasks. Ourimplementation is available at https://github.com/jw9730/hot.

Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs