dc.contributor.author	Villanueva Aylagas, Monica	en_US
dc.contributor.author	Anadon Leon, Hector	en_US
dc.contributor.author	Teye, Mattias	en_US
dc.contributor.author	Tollmar, Konrad	en_US
dc.contributor.editor	Dominik L. Michels	en_US
dc.contributor.editor	Soeren Pirk	en_US
dc.date.accessioned	2022-08-10T15:19:54Z
dc.date.available	2022-08-10T15:19:54Z
dc.date.issued	2022
dc.description.abstract	We present Voice2Face: a Deep Learning model that generates face and tongue animations directly from recorded speech. Our approach consists of two steps: a conditional Variational Autoencoder generates mesh animations from speech, while a separate module maps the animations to rig controller space. Our contributions include an automated method for speech style control, a method to train a model with data from multiple quality levels, and a method for animating the tongue. Unlike previous works, our model generates animations without speaker-dependent characteristics while allowing speech style control. We demonstrate through a user study that Voice2Face significantly outperforms a comparative state-of-the-art model in terms of perceived animation quality, and our quantitative evaluation suggests that Voice2Face yields more accurate lip closure in speech with bilabials through our speech style optimization. Both evaluations also show that our data quality conditioning scheme outperforms both an unconditioned model and a model trained with a smaller high-quality dataset. Finally, the user study shows a preference for animations including tongue. Results from our model can be seen at https://go.ea.com/voice2face.	en_US
dc.description.number	8
dc.description.sectionheaders	Capture, Tracking, and Facial Animation
dc.description.seriesinformation	Computer Graphics Forum
dc.description.volume	41
dc.identifier.doi	10.1111/cgf.14640
dc.identifier.issn	1467-8659
dc.identifier.pages	255-265
dc.identifier.pages	11 pages
dc.identifier.uri	https://doi.org/10.1111/cgf.14640
dc.identifier.uri	https://diglib.eg.org:443/handle/10.1111/cgf14640
dc.publisher	The Eurographics Association and John Wiley & Sons Ltd.	en_US
dc.subject	CCS Concepts: Computing methodologies --> Animation; Neural networks; Latent variable models; Learning latent representations; Additional Key Words and Phrases: Deep Learning, Facial animation, Tongue animation, Lip synchronization, Rig animation
dc.subject	Computing methodologies
dc.subject	Animation
dc.subject	Neural networks
dc.subject	Latent variable models
dc.subject	Learning latent representations
dc.subject	Additional Key Words and Phrases
dc.subject	Deep Learning
dc.subject	Facial animation
dc.subject	Tongue animation
dc.subject	Lip synchronization
dc.subject	Rig animation
dc.title	Voice2Face: Audio-driven Facial and Tongue Rig Animations with cVAEs	en_US

Voice2Face: Audio-driven Facial and Tongue Rig Animations with cVAEs

Files

Original bundle

Collections