English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Vision-Language Navigation (VLN) is a challenging task that requires anembodied agent to perform action-level modality alignment, i.e., makeinstruction-asked actions sequentially in complex visual environments. Mostexisting VLN agents learn the instruction-path data directly and cannotsufficiently explore action-level alignment knowledge inside the multi-modalinputs. In this paper, we propose modAlity-aligneD Action PrompTs (ADAPT),which provides the VLN agent with action prompts to enable the explicitlearning of action-level modality alignment to pursue successful navigation.Specifically, an action prompt is defined as a modality-aligned pair of animage sub-prompt and a text sub-prompt, where the former is a single-viewobservation and the latter is a phrase like ''walk past the chair''. Whenstarting navigation, the instruction-related action prompt set is retrievedfrom a pre-built action prompt base and passed through a prompt encoder toobtain the prompt feature. Then the prompt feature is concatenated with theoriginal instruction feature and fed to a multi-layer transformer for actionprediction. To collect high-quality action prompts into the prompt base, we usethe Contrastive Language-Image Pretraining (CLIP) model which has powerfulcross-modality alignment ability. A modality alignment loss and a sequentialconsistency loss are further introduced to enhance the alignment of the actionprompt and enforce the agent to focus on the related prompt sequentially.Experimental results on both R2R and RxR show the superiority of ADAPT overstate-of-the-art methods.

ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts