English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A major challenge in visually grounded language generation is to build robustbenchmark datasets and models that can generalize well in real-world settings.To do this, it is critical to ensure that our evaluation protocols are correct,and benchmarks are reliable. In this work, we set forth to design a set ofexperiments to understand an important but often ignored problem in visuallygrounded language generation: given that humans have different utilities andvisual attention, how will the sample variance in multi-reference datasetsaffect the models' performance? Empirically, we study several multi-referencedatasets and corresponding vision-and-language tasks. We show that it is ofparamount importance to report variance in experiments; that human-generatedreferences could vary drastically in different datasets/tasks, revealing thenature of each task; that metric-wise, CIDEr has shown systematically largervariances than others. Our evaluations on reference-per-instance shed light onthe design of reliable datasets in the future.

Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations