English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Capturing the interactions between humans and their environment in 3D isimportant for many applications in robotics, graphics, and vision. Recent worksto reconstruct the 3D human and object from a single RGB image do not haveconsistent relative translation across frames because they assume a fixeddepth. Moreover, their performance drops significantly when the object isoccluded. In this work, we propose a novel method to track the 3D human,object, contacts between them, and their relative translation across framesfrom a single RGB camera, while being robust to heavy occlusions. Our method isbuilt on two key insights. First, we condition our neural field reconstructionsfor human and object on per-frame SMPL model estimates obtained by pre-fittingSMPL to a video sequence. This improves neural reconstruction accuracy andproduces coherent relative translation across frames. Second, human and objectmotion from visible frames provides valuable information to infer the occludedobject. We propose a novel transformer-based neural network that explicitlyuses object visibility and human motion to leverage neighbouring frames to makepredictions for the occluded frames. Building on these insights, our method isable to track both human and object robustly even under occlusions. Experimentson two datasets show that our method significantly improves over thestate-of-the-art methods. Our code and pretrained models are available at:https://virtualhumans.mpi-inf.mpg.de/VisTracker

Visibility Aware Human-Object Interaction Tracking from Single RGB Camera