English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision language decision making (VLDM) is a challenging multimodal task. Theagent have to understand complex human instructions and complete compositionaltasks involving environment navigation and object manipulation. However, thelong action sequences involved in VLDM make the task difficult to learn. Froman environment perspective, we find that task episodes can be divided intofine-grained \textit{units}, each containing a navigation phase and aninteraction phase. Since the environment within a unit stays unchanged, wepropose a novel hybrid-training framework that enables active exploration inthe environment and reduces the exposure bias. Such framework leverages theunit-grained configurations and is model-agnostic. Specifically, we design aUnit-Transformer (UT) with an intrinsic recurrent state that maintains aunit-scale cross-modal memory. Through extensive experiments on the TEACHbenchmark, we demonstrate that our proposed framework outperforms existingstate-of-the-art methods in terms of all evaluation metrics. Overall, our workintroduces a novel approach to tackling the VLDM task by breaking it down intosmaller, manageable units and utilizing a hybrid-training framework. By doingso, we provide a more flexible and effective solution for multimodal decisionmaking.

Breaking Down the Task: A Unit-Grained Hybrid Training Framework for Vision and Language Decision Making