English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

For the past ten years, CNN has reigned supreme in the world of computervision, but recently, Transformer has been on the rise. However, the quadraticcomputational cost of self-attention has become a serious problem in practiceapplications. There has been much research on architectures without CNN andself-attention in this context. In particular, MLP-Mixer is a simplearchitecture designed using MLPs and hit an accuracy comparable to the VisionTransformer. However, the only inductive bias in this architecture is theembedding of tokens. This leaves open the possibility of incorporating anon-convolutional (or non-local) inductive bias into the architecture, so weused two simple ideas to incorporate inductive bias into the MLP-Mixer whiletaking advantage of its ability to capture global correlations. A way is todivide the token-mixing block vertically and horizontally. Another way is tomake spatial correlations denser among some channels of token-mixing. With thisapproach, we were able to improve the accuracy of the MLP-Mixer while reducingits parameters and computational complexity. The small model that is RaftMLP-Sis comparable to the state-of-the-art global MLP-based model in terms ofparameters and efficiency per calculation. In addition, we tackled the problemof fixed input image resolution for global MLP-based models by utilizingbicubic interpolation. We demonstrated that these models could be applied asthe backbone of architectures for downstream tasks such as object detection.However, it did not have significant performance and mentioned the need forMLP-specific architectures for downstream tasks for global MLP-based models.The source code in PyTorch version is available at\url{https://github.com/okojoalg/raft-mlp}.

RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?