English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

The task of image captioning aims to generate captions directly from imagesvia the automatically learned cross-modal generator. To build a well-performinggenerator, existing approaches usually need a large number of described images,which requires a huge effects on manual labeling. However, in real-worldapplications, a more general scenario is that we only have limited amount ofdescribed images and a large number of undescribed images. Therefore, aresulting challenge is how to effectively combine the undescribed images intothe learning of cross-modal generator. To solve this problem, we propose anovel image captioning method by exploiting the Cross-modal Prediction andRelation Consistency (CPRC), which aims to utilize the raw image input toconstrain the generated sentence in the commonly semantic space. In detail,considering that the heterogeneous gap between modalities always leads to thesupervision difficulty of using the global embedding directly, CPRC turns totransform both the raw image and corresponding generated sentence into theshared semantic space, and measure the generated sentence from two aspects: 1)Prediction consistency. CPRC utilizes the prediction of raw image as soft labelto distill useful supervision for the generated sentence, rather than employingthe traditional pseudo labeling; 2) Relation consistency. CPRC develops a novelrelation consistency between augmented images and corresponding generatedsentences to retain the important relational knowledge. In result, CPRCsupervises the generated sentence from both the informativeness andrepresentativeness perspectives, and can reasonably use the undescribed imagesto learn a more effective generator under the semi-supervised scenario.

Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning