English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In this paper, we propose an autonomous information seeking visual questionanswering framework, AVIS. Our method leverages a Large Language Model (LLM) todynamically strategize the utilization of external tools and to investigatetheir outputs, thereby acquiring the indispensable knowledge needed to provideanswers to the posed questions. Responding to visual questions that necessitateexternal knowledge, such as "What event is commemorated by the buildingdepicted in this image?", is a complex task. This task presents a combinatorialsearch space that demands a sequence of actions, including invoking APIs,analyzing their responses, and making informed decisions. We conduct a userstudy to collect a variety of instances of human decision-making when facedwith this task. This data is then used to design a system comprised of threecomponents: an LLM-powered planner that dynamically determines which tool touse next, an LLM-powered reasoner that analyzes and extracts key informationfrom the tool outputs, and a working memory component that retains the acquiredinformation throughout the process. The collected user behavior serves as aguide for our system in two key ways. First, we create a transition graph byanalyzing the sequence of decisions made by users. This graph delineatesdistinct states and confines the set of actions available at each state.Second, we use examples of user decision-making to provide our LLM-poweredplanner and reasoner with relevant contextual instances, enhancing theircapacity to make informed decisions. We show that AVIS achievesstate-of-the-art results on knowledge-intensive visual question answeringbenchmarks such as Infoseek and OK-VQA.

AVIS: Autonomous Visual Information Seeking with Large Language Model Agent