English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Recent progress in self-supervised (SSL) visual representation learning hasled to the development of several different proposed frameworks that rely onaugmentations of images but use different loss functions. However, there arefew theoretically grounded principles to guide practice, so practicalimplementation of each SSL framework requires several heuristics to achievecompetitive performance. In this work, we build on recent analytical results todesign practical recommendations for competitive and efficient SSL that aregrounded in theory. Specifically, recent theory tells us that existing SSLframeworks are minimizing the same idealized loss, which is to learn featuresthat best match the data similarity kernel defined by the augmentations used.We show how this idealized loss can be reformulated to a functionallyequivalent loss that is more efficient to compute. We study the implicit biasof using gradient descent to minimize our reformulated loss function and findthat using a stronger orthogonalization constraint with a reduced projectordimensionality should yield good representations. Furthermore, the theory tellsus that approximating the reformulated loss should be improved by increasingthe number of augmentations, and as such using multiple augmentations shouldlead to improved convergence. We empirically verify our findings on CIFAR, STLand Imagenet datasets, wherein we demonstrate an improved linear readoutperformance when training a ResNet-backbone using our theoretically groundedrecommendations. Remarkably, we also demonstrate that by leveraging theseinsights, we can reduce the pretraining dataset size by up to 2$\times$ whilemaintaining downstream accuracy simply by using more data augmentations. Takentogether, our work provides theoretically grounded recommendations that can beused to improve SSL convergence and efficiency.

Harnessing small projectors and multiple views for efficient vision pretraining