English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Vision-and-Language Navigation (VLN) is the task that requires an agent tonavigate through the environment based on natural language instructions. Ateach step, the agent takes the next action by selecting from a set of navigablelocations. In this paper, we aim to take one step further and explore whetherthe agent can benefit from generating the potential future view duringnavigation. Intuitively, humans will have an expectation of how the futureenvironment will look like, based on the natural language instructions andsurrounding views, which will aid correct navigation. Hence, to equip the agentwith this ability to generate the semantics of future navigation views, wefirst propose three proxy tasks during the agent's in-domain pre-training:Masked Panorama Modeling (MPM), Masked Trajectory Modeling (MTM), and ActionPrediction with Image Generation (APIG). These three objectives teach the modelto predict missing views in a panorama (MPM), predict missing steps in the fulltrajectory (MTM), and generate the next view based on the full instruction andnavigation history (APIG), respectively. We then fine-tune the agent on the VLNtask with an auxiliary loss that minimizes the difference between the viewsemantics generated by the agent and the ground truth view semantics of thenext step. Empirically, our VLN-SIG achieves the new state-of-the-art on boththe Room-to-Room dataset and the CVDN dataset. We further show that our agentlearns to fill in missing patches in future views qualitatively, which bringsmore interpretability over agents' predicted actions. Lastly, we demonstratethat learning to predict future view semantics also enables the agent to havebetter performance on longer paths.

Improving Vision-and-Language Navigation by Generating Future-View Image Semantics