English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretraining has become a standard technique in computer vision and naturallanguage processing, which usually helps to improve performance substantially.Previously, the most dominant pretraining method is transfer learning (TL),which uses labeled data to learn a good representation network. Recently, a newpretraining approach -- self-supervised learning (SSL) -- has demonstratedpromising results on a wide range of applications. SSL does not requireannotated labels. It is purely conducted on input data by solving auxiliarytasks defined on the input data examples. The current reported results showthat in certain applications, SSL outperforms TL and the other way around inother applications. There has not been a clear understanding on what propertiesof data and tasks render one approach outperforms the other. Without aninformed guideline, ML researchers have to try both methods to find out whichone is better empirically. It is usually time-consuming to do so. In this work,we aim to address this problem. We perform a comprehensive comparative studybetween SSL and TL regarding which one works better under different propertiesof data and tasks, including domain difference between source and target tasks,the amount of pretraining data, class imbalance in source data, and usage oftarget data for additional pretraining, etc. The insights distilled from ourcomparative studies can help ML researchers decide which method to use based onthe properties of their applications.

Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms