English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Leveraging massive knowledge and learning schemes from large language models(LLMs), recent machine learning models show notable successes in buildinggeneralist agents that exhibit the capability of general-purpose task solvingin diverse domains, including natural language processing, computer vision, androbotics. However, a significant challenge remains as these models exhibitlimited ability in understanding and interacting with the 3D world. We arguethis limitation significantly hinders the current models from performingreal-world tasks and further achieving general intelligence. To this end, weintroduce an embodied multi-modal and multi-task generalist agent that excelsin perceiving, grounding, reasoning, planning, and acting in the 3D world. Ourproposed agent, referred to as LEO, is trained with shared LLM-based modelarchitectures, objectives, and weights in two stages: (i) 3D vision-languagealignment and (ii) 3D vision-language-action instruction tuning. To facilitatethe training, we meticulously curate and generate an extensive datasetcomprising object-level and scene-level multi-modal tasks with exceeding scaleand complexity, necessitating a deep understanding of and interaction with the3D world. Through rigorous experiments, we demonstrate LEO's remarkableproficiency across a wide spectrum of tasks, including 3D captioning, questionanswering, embodied reasoning, embodied navigation, and robotic manipulation.Our ablation results further provide valuable insights for the development offuture embodied generalist agents.

An Embodied Generalist Agent in 3D World