English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

To achieve human-level dexterity, robots must infer spatial awareness frommultimodal sensing to reason over contact interactions. During in-handmanipulation of novel objects, such spatial awareness involves estimating theobject's pose and shape. The status quo for in-hand perception primarilyemploys vision, and restricts to tracking a priori known objects. Moreover,visual occlusion of objects in-hand is imminent during manipulation, preventingcurrent systems to push beyond tasks without occlusion. We combine vision andtouch sensing on a multi-fingered hand to estimate an object's pose and shapeduring in-hand manipulation. Our method, NeuralFeels, encodes object geometryby learning a neural field online and jointly tracks it by optimizing a posegraph problem. We study multimodal in-hand perception in simulation and thereal-world, interacting with different objects via a proprioception-drivenpolicy. Our experiments show final reconstruction F-scores of $81$% and averagepose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with knownCAD models. Additionally, we observe that under heavy visual occlusion we canachieve up to $94$% improvements in tracking compared to vision-only methods.Our results demonstrate that touch, at the very least, refines and, at the verybest, disambiguates visual estimates during in-hand manipulation. We releaseour evaluation dataset of 70 experiments, FeelSight, as a step towardsbenchmarking in this domain. Our neural representation driven by multimodalsensing can serve as a perception backbone towards advancing robot dexterity.Videos can be found on our project websitehttps://suddhu.github.io/neural-feels/

Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation