English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The field of AI agents is advancing at an unprecedented rate due to thecapabilities of large language models (LLMs). However, LLM-driven visual agentsmainly focus on solving tasks for the image modality, which limits theirability to understand the dynamic nature of the real world, making it still farfrom real-life applications, e.g., guiding students in laboratory experimentsand identifying their mistakes. Considering the video modality better reflectsthe ever-changing and perceptually intensive nature of real-world scenarios, wedevise DoraemonGPT, a comprehensive and conceptually elegant system driven byLLMs to handle dynamic video tasks. Given a video with a question/task,DoraemonGPT begins by converting the input video with massive content into asymbolic memory that stores \textit{task-related} attributes. This structuredrepresentation allows for spatial-temporal querying and reasoning by sub-tasktools, resulting in concise and relevant intermediate results. Recognizing thatLLMs have limited internal knowledge when it comes to specialized domains(e.g., analyzing the scientific principles underlying experiments), weincorporate plug-and-play tools to assess external knowledge and address tasksacross different domains. Moreover, we introduce a novel LLM-driven plannerbased on Monte Carlo Tree Search to efficiently explore the large planningspace for scheduling various tools. The planner iteratively finds feasiblesolutions by backpropagating the result's reward, and multiple solutions can besummarized into an improved final answer. We extensively evaluate DoraemonGPTin dynamic scenes and provide in-the-wild showcases demonstrating its abilityto handle more complex questions than previous studies.

DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models