English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The current literature on self-supervised learning (SSL) focuses ondeveloping learning objectives to train neural networks more effectively onunlabeled data. The typical development process involves takingwell-established architectures, e.g., ResNet demonstrated on ImageNet, andusing them to evaluate newly developed objectives on downstream scenarios.While convenient, this does not take into account the role of architectureswhich has been shown to be crucial in the supervised learning literature. Inthis work, we establish extensive empirical evidence showing that a networkarchitecture plays a significant role in SSL. We conduct a large-scale studywith over 100 variants of ResNet and MobileNet architectures and evaluate themacross 11 downstream scenarios in the SSL setting. We show that there is no onenetwork that performs consistently well across the scenarios. Based on this, wepropose to learn not only network weights but also architecture topologies inthe SSL regime. We show that "self-supervised architectures" outperform popularhandcrafted architectures (ResNet18 and MobileNetV2) while performingcompetitively with the larger and computationally heavy ResNet50 on major imageclassification benchmarks (ImageNet-1K, iNat2021, and more). Our resultssuggest that it is time to consider moving beyond handcrafted architectures inSSL and start thinking about incorporating architecture search intoself-supervised learning objectives.

One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning