English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Most approaches for self-supervised learning (SSL) are optimised on curatedbalanced datasets, e.g. ImageNet, despite the fact that natural data usuallyexhibits long-tail distributions. In this paper, we analyse the behaviour ofone of the most popular variants of SSL, i.e. contrastive methods, on long-taildata. In particular, we investigate the role of the temperature parameter$\tau$ in the contrastive loss, by analysing the loss through the lens ofaverage distance maximisation, and find that a large $\tau$ emphasisesgroup-wise discrimination, whereas a small $\tau$ leads to a higher degree ofinstance discrimination. While $\tau$ has thus far been treated exclusively asa constant hyperparameter, in this work, we propose to employ a dynamic $\tau$and show that a simple cosine schedule can yield significant improvements inthe learnt representations. Such a schedule results in a constant `taskswitching' between an emphasis on instance discrimination and group-wisediscrimination and thereby ensures that the model learns both group-wisefeatures, as well as instance-specific details. Since frequent classes benefitfrom the former, while infrequent classes require the latter, we find thismethod to consistently improve separation between the classes in long-tail datawithout any additional computational cost.

Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data