English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

With the rise of Transformers as the standard for language processing, andtheir advancements in computer vision, there has been a corresponding growth inparameter size and amounts of training data. Many have come to believe thatbecause of this, transformers are not suitable for small sets of data. Thistrend leads to concerns such as: limited availability of data in certainscientific domains and the exclusion of those with limited resource fromresearch in the field. In this paper, we aim to present an approach forsmall-scale learning by introducing Compact Transformers. We show for the firsttime that with the right size, convolutional tokenization, transformers canavoid overfitting and outperform state-of-the-art CNNs on small datasets. Ourmodels are flexible in terms of model size, and can have as little as 0.28Mparameters while achieving competitive results. Our best model can reach 98%accuracy when training from scratch on CIFAR-10 with only 3.7M parameters,which is a significant improvement in data-efficiency over previous Transformerbased models being over 10x smaller than other transformers and is 15% the sizeof ResNet50 while achieving similar performance. CCT also outperforms manymodern CNN based approaches, and even some recent NAS-based approaches.Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracywith 29% as many parameters as ViT), as well as NLP tasks. Our simple andcompact design for transformers makes them more feasible to study for thosewith limited computing resources and/or dealing with small datasets, whileextending existing research efforts in data efficient transformers. Our codeand pre-trained models are publicly available athttps://github.com/SHI-Labs/Compact-Transformers.

Escaping the Big Data Paradigm with Compact Transformers