English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Prompt-based learning has been demonstrated as a compelling paradigmcontributing to large language models' tremendous success (LLMs). Inspired bytheir success in language tasks, existing research has leveraged LLMs inembodied instruction following and task planning. However, not much attentionhas been paid to embodied tasks with multimodal prompts, combining visionsignals with text descriptions. This type of task poses a major challenge torobots' capability to understand the interconnection and complementaritybetween vision and language signals. In this work, we introduce an effectiveframework that learns a policy to perform robot manipulation with multimodalprompts from multi-task expert trajectories. Our methods consist of a two-stagetraining pipeline that performs inverse dynamics pretraining and multi-taskfinetuning. To facilitate multimodal understanding, we design our multimodalprompt encoder by augmenting a pretrained LM with a residual connection to thevisual input and model the dependencies among action dimensions. Empirically,we evaluate the efficacy of our method on the VIMA-BENCH and establish a newstate-of-the-art (10% improvement in success rate). Moreover, we demonstratethat our model exhibits remarkable in-context learning ability.

Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning