English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Significant progress has been made on visual captioning, largely relying onpre-trained features and later fixed object detectors that serve as rich inputsto auto-regressive models. A key limitation of such methods, however, is thatthe output of the model is conditioned only on the object detector's outputs.The assumption that such outputs can represent all necessary information isunrealistic, especially when the detector is transferred across datasets. Inthis work, we reason about the graphical model induced by this assumption, andpropose to add an auxiliary input to represent missing information such asobject relationships. We specifically propose to mine attributes andrelationships from the Visual Genome dataset and condition the captioning modelon them. Crucially, we propose (and show to be important) the use of amulti-modal pre-trained model (CLIP) to retrieve such contextual descriptions.Further, object detector models are frozen and do not have sufficient richnessto allow the captioning model to properly ground them. As a result, we proposeto condition both the detector and description outputs on the image, and showqualitatively and quantitatively that this can improve grounding. We validateour method on image captioning, perform thorough analyses of each component andimportance of the pre-trained multi-modal model, and demonstrate significantimprovements over the current state of the art, specifically +7.5% in CIDEr and+1.3% in BLEU-4 metrics.

Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning