English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose DisCo-CLIP, a distributed memory-efficient CLIP training approach,to reduce the memory consumption of contrastive loss when training contrastivelearning models. Our approach decomposes the contrastive loss and its gradientcomputation into two parts, one to calculate the intra-GPU gradients and theother to compute the inter-GPU gradients. According to our decomposition, onlythe intra-GPU gradients are computed on the current GPU, while the inter-GPUgradients are collected via all_reduce from other GPUs instead of beingrepeatedly computed on every GPU. In this way, we can reduce the GPU memoryconsumption of contrastive loss computation from $\bigO(B^2)$ to$\bigO(\frac{B^2}{N})$, where $B$ and $N$ are the batch size and the number ofGPUs used for training. Such a distributed solution is mathematicallyequivalent to the original non-distributed contrastive loss computation,without sacrificing any computation accuracy. It is particularly efficient forlarge-batch CLIP training. For instance, DisCo-CLIP can enable contrastivetraining of a ViT-B/32 model with a batch size of 32K or 196K using 8 or 64A100 40GB GPUs, compared with the original CLIP solution which requires 128A100 40GB GPUs to train a ViT-B/32 model with a batch size of 32K. The codewill be released at https://github.com/IDEA-Research/DisCo-CLIP

DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training