Semantics-aware Motion Retargeting with Vision-Language Models

Capturing and preserving motion semantics is essential to motion retargetingbetween animation characters. However, most of the previous works neglect thesemantic information or rely on human-designed joint-level representations.Here, we present a novel Semantics-aware Motion reTargeting (SMT) method withthe advantage of vision-language models to extract and maintain meaningfulmotion semantics. We utilize a differentiable module to render 3D motions. Thenthe high-level motion semantics are incorporated into the motion retargetingprocess by feeding the vision-language model with the rendered images andaligning the extracted semantic embeddings. To ensure the preservation offine-grained motion details and high-level semantics, we adopt a two-stagepipeline consisting of skeleton-aware pre-training and fine-tuning withsemantics and geometry constraints. Experimental results show the effectivenessof the proposed method in producing high-quality motion retargeting resultswhile accurately preserving motion semantics. Project page can be found athttps://sites.google.com/view/smtnet.