English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In recent years, self-attention has become the dominant paradigm for sequencemodeling in a variety of domains. However, in domains with very long sequencelengths the $\mathcal{O}(T^2)$ memory and $\mathcal{O}(T^2 H)$ compute costscan make using transformers infeasible. Motivated by problems in malwaredetection, where sequence lengths of $T \geq 100,000$ are a roadblock to deeplearning, we re-cast self-attention using the neuro-symbolic approach ofHolographic Reduced Representations (HRR). In doing so we perform the samehigh-level strategy of the standard self-attention: a set of queries matchingagainst a set of keys, and returning a weighted response of the values for eachkey. Implemented as a ``Hrrformer'' we obtain several benefits including$\mathcal{O}(T H \log H)$ time complexity, $\mathcal{O}(T H)$ space complexity,and convergence in $10\times$ fewer epochs. Nevertheless, the Hrrformerachieves near state-of-the-art accuracy on LRA benchmarks and we are able tolearn with just a single layer. Combined, these benefits make our Hrrformer thefirst viable Transformer for such long malware classification sequences and upto $280\times$ faster to train on the Long Range Arena benchmark. Code isavailable at\url{https://github.com/NeuromorphicComputationResearchProgram/Hrrformer}

Recasting Self-Attention with Holographic Reduced Representations