English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Humans rely on the synergy of their senses for most essential tasks. Fortasks requiring object manipulation, we seamlessly and effectively exploit thecomplementarity of our senses of vision and touch. This paper draws inspirationfrom such capabilities and aims to find a systematic approach to fuse visualand tactile information in a reinforcement learning setting. We propose MaskedMultimodal Learning (M3L), which jointly learns a policy and visual-tactilerepresentations based on masked autoencoding. The representations jointlylearned from vision and touch improve sample efficiency, and unlockgeneralization capabilities beyond those achievable through each of the sensesseparately. Remarkably, representations learned in a multimodal setting alsobenefit vision-only policies at test time. We evaluate M3L on three simulatedenvironments with both visual and tactile observations: robotic insertion, dooropening, and dexterous in-hand manipulation, demonstrating the benefits oflearning a multimodal policy. Code and videos of the experiments are availableat https://sferrazza.cc/m3l_site.

The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning