English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Previous studies dominantly target at self-supervised learning on real-valuednetworks and have achieved many promising results. However, on the morechallenging binary neural networks (BNNs), this task has not yet been fullyexplored in the community. In this paper, we focus on this more difficultscenario: learning networks where both weights and activations are binary,meanwhile, without any human annotated labels. We observe that the commonlyused contrastive objective is not satisfying on BNNs for competitive accuracy,since the backbone network contains relatively limited capacity andrepresentation ability. Hence instead of directly applying existingself-supervised methods, which cause a severe decline in performance, wepresent a novel guided learning paradigm from real-valued to distill binarynetworks on the final prediction distribution, to minimize the loss and obtaindesirable accuracy. Our proposed method can boost the simple contrastivelearning baseline by an absolute gain of 5.5~15% on BNNs. We further revealthat it is difficult for BNNs to recover the similar predictive distributionsas real-valued models when training without labels. Thus, how to calibrate themis key to address the degradation in performance. Extensive experiments areconducted on the large-scale ImageNet and downstream datasets. Our methodachieves substantial improvement over the simple contrastive learning baseline,and is even comparable to many mainstream supervised BNN methods. Code isavailable at https://github.com/szq0214/S2-BNN.

S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration