English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recently, reference-free metrics such as CLIPScore (Hessel et al., 2021) andUMIC (Lee et al., 2021) have been proposed for automatic evaluation of imagecaptions, demonstrating a high correlation with human judgment. In this work,our focus lies in evaluating the robustness of these metrics in scenarios thatrequire distinguishing between two captions with high lexical overlap but verydifferent meanings. Our findings reveal that despite their high correlationwith human judgment, both CLIPScore and UMIC struggle to identify fine-grainederrors in captions. However, when comparing different types of fine-grainederrors, both metrics exhibit limited sensitivity to implausibility of captionsand strong sensitivity to lack of sufficient visual grounding. Probing furtherinto the visual grounding aspect, we found that both CLIPScore and UMIC areimpacted by the size of image-relevant objects mentioned in the caption, andthat CLIPScore is also sensitive to the number of mentions of image-relevantobjects in the caption. In terms of linguistic aspects of a caption, we foundthat both metrics lack the ability to comprehend negation, UMIC is sensitive tocaption lengths, and CLIPScore is insensitive to the structure of the sentence.We hope our findings will serve as a valuable guide towards improvingreference-free evaluation in image captioning.

An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics