Open AccessEvaluating Visual Reasoning Through Grounded Language Understanding
Author(s) -
Suhr Alane,
Lewis Mike,
Yeh James,
Artzi Yoav
Publication year - 2018
Publication title -
ai magazine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.597
H-Index - 79
eISSN - 2371-9621
pISSN - 0738-4602
DOI - 10.1609/aimag.v39i2.2796
Subject(s) - computer science , sentence , artificial intelligence , natural language , natural language understanding , natural language processing , set (abstract data type) , task (project management) , benchmark (surveying) , visual reasoning , simple (philosophy) , programming language , philosophy , management , geodesy , epistemology , economics , geography
Autonomous systems that understand natural language must reason about complex language and visual observations. Key to making progress toward such systems is the availability of benchmark data sets and tasks. We introduce the Cornell Natural Language Visual Reasoning (NLVR) corpus, which targets reasoning skills like counting, comparisons, and set theory. NLVR contains 92,244 examples of natural language statements paired with synthetic images and annotated with Boolean values for the simple task of determining whether the sentence is true or false about the image. While it presents a simple task, NLVR has been developed to challenge systems with diverse linguistic phenomena and complex reasoning. Linguistic analysis confirms that NLVR presents diversity and complexity beyond what is provided by contemporary benchmarks. Empirical evaluation of several methods further demonstrates the open challenges NLVR presents.