English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Grounding-based vision and language models have been successfully applied tolow-level vision tasks, aiming to precisely locate objects referred incaptions. The effectiveness of grounding representation learning heavily relieson the scale of the training dataset. Despite being a useful data enrichmentstrategy, data augmentation has received minimal attention in existing visionand language tasks as augmentation for image-caption pairs is non-trivial. Inthis study, we propose a robust phrase grounding model trained withtext-conditioned and text-unconditioned data augmentations. Specifically, weapply text-conditioned color jittering and horizontal flipping to ensuresemantic consistency between images and captions. To guarantee image-captioncorrespondence in the training samples, we modify the captions according topre-defined keywords when applying horizontal flipping. Additionally, inspiredby recent masked signal reconstruction, we propose to use pixel-level maskingas a novel form of data augmentation. While we demonstrate our dataaugmentation method with MDETR framework, the proposed approach is applicableto common grounding-based vision and language tasks with other frameworks.Finally, we show that image encoder pretrained on large-scale image andlanguage datasets (such as CLIP) can further improve the results. Throughextensive experiments on three commonly applied datasets: Flickr30k, referringexpressions and GQA, our method demonstrates advanced performance over thestate-of-the-arts with various metrics. Code can be found inhttps://github.com/amzn/augment-the-pairs-wacv2024.

Augment the Pairs: Semantics-Preserving Image-Caption Pair Augmentation for Grounding-Based Vision and Language Models