English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

How should we learn visual representations for embodied agents that must seeand move? The status quo is tabula rasa in vivo, i.e. learning visualrepresentations from scratch while also learning to move, potentially augmentedwith auxiliary tasks (e.g. predicting the action taken between two successiveobservations). In this paper, we show that an alternative 2-stage strategy isfar more effective: (1) offline pretraining of visual representations withself-supervised learning (SSL) using large-scale pre-rendered images of indoorenvironments (Omnidata), and (2) online finetuning of visuomotorrepresentations on specific tasks with image augmentations under long learningschedules. We call this method Offline Visual Representation Learning (OVRL).We conduct large-scale experiments - on 3 different 3D datasets (Gibson, HM3D,MP3D), 2 tasks (ImageNav, ObjectNav), and 2 policy learning algorithms (RL, IL)- and find that the OVRL representations lead to significant across-the-boardimprovements in state of art, on ImageNav from 29.2% to 54.2% (+25% absolute,86% relative) and on ObjectNav from 18.1% to 23.2% (+5.1% absolute, 28%relative). Importantly, both results were achieved by the same visual encodergeneralizing to datasets that were not seen during pretraining. While thebenefits of pretraining sometimes diminish (or entirely disappear) with longfinetuning schedules, we find that OVRL's performance gains continue toincrease (not decrease) as the agent is trained for 2 billion frames ofexperience.

Offline Visual Representation Learning for Embodied Navigation