Physics-based Motion Retargeting from Sparse Inputs

Reda, Daniele; Won, Jungdam; Ye, Yuting; Panne, Michiel van de; Winkler, Alexander

Physics-based Motion Retargeting from Sparse Inputs

Date

2023

Authors

Reda, Daniele
Won, Jungdam
Ye, Yuting
Panne, Michiel van de
Winkler, Alexander

Publisher

ACM Association for Computing Machinery

Abstract

Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user’s motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user’s pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time.We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human.We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

CCS Concepts: Computing methodologies -> Reinforcement learning; Physical simulation retargeting, reinforcement learning, physics-based simulation, computer animation"

        @inproceedings{10.1145:3606928
,
booktitle = {Proceedings of the ACM on Computer Graphics and Interactive Techniques
},
editor = {Wang, Huamin and 
Ye, Yuting and 
Victor Zordan
},
title = {{Physics-based Motion Retargeting from Sparse Inputs
}},
author = {Reda, Daniele and 
Won, Jungdam and 
Ye, Yuting and 
Panne, Michiel van de and 
Winkler, Alexander
},
year = {2023
},
publisher = {ACM Association for Computing Machinery
},
ISSN = {2577-6193
},
ISBN = {},
DOI = {10.1145/3606928
}
}

URI

https://doi.org/10.1145/3606928
https://diglib.eg.org:443/handle/10.1145/3606928

Collections

SCA 2023: Eurographics/SIGGRAPH Symposium on Computer Animation

Full item page