English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We aim to teach robots to perform simple object manipulation tasks bywatching a single video demonstration. Towards this goal, we propose anoptimization approach that outputs a coarse and temporally evolving 3D scene tomimic the action demonstrated in the input video. Similar to previous work, adifferentiable renderer ensures perceptual fidelity between the 3D scene andthe 2D video. Our key novelty lies in the inclusion of a differentiableapproach to solve a set of Ordinary Differential Equations (ODEs) that allowsus to approximately model laws of physics such as gravity, friction, andhand-object or object-object interactions. This not only enables us todramatically improve the quality of estimated hand and object states, but alsoproduces physically admissible trajectories that can be directly translated toa robot without the need for costly reinforcement learning. We evaluate ourapproach on a 3D reconstruction task that consists of 54 video demonstrationssourced from 9 actions such as pull something from right to left or putsomething in front of something. Our approach improves over previousstate-of-the-art by almost 30%, demonstrating superior quality on especiallychallenging actions involving physical interactions of two objects such as putsomething onto something. Finally, we showcase the learned skills on a FrankaEmika Panda robot.

Learning Object Manipulation Skills from Video via Approximate Differentiable Physics