English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large-scale pre-trained multi-modal models (e.g., CLIP) demonstrate strongzero-shot transfer capability in many discriminative tasks. Their adaptation tozero-shot image-conditioned text generation tasks has drawn increasinginterest. Prior arts approach to zero-shot captioning by either utilizing theexisting large language models (e.g., GPT-2) or pre-training theencoder-decoder network in an end-to-end manner. In this work, we propose asimple framework, named DeCap, for zero-shot captioning. We introduce alightweight visual-aware language decoder. This decoder is both data-efficientand computation-efficient: 1) it only requires the text data for training,easing the burden on the collection of paired data. 2) it does not requireend-to-end training. When trained with text-only data, the decoder takes thetext embedding extracted from the off-the-shelf CLIP encoder as a prefixembedding. The challenge is that the decoder is trained on the text corpus butat the inference stage, it needs to generate captions based on visual inputs.The modality gap issue is widely observed in multi-modal contrastive modelsthat prevents us from directly taking the visual embedding as the prefixembedding. We propose a training-free mechanism to reduce the modality gap. Weproject the visual embedding into the CLIP text embedding space, while theprojected embedding retains the information of the visual input. Taking theprojected embedding as the prefix embedding, the decoder generates high-qualitydescriptions that match the visual input. The experiments show that DeCapoutperforms other zero-shot captioning methods and unpaired captioning methodson the typical image captioning benchmarks, i.e., MSCOCO and NoCaps.

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training