Villanueva Aylagas, MonicaAnadon Leon, HectorTeye, MattiasTollmar, KonradDominik L. MichelsSoeren Pirk2022-08-102022-08-1020221467-8659https://doi.org/10.1111/cgf.14640https://diglib.eg.org:443/handle/10.1111/cgf14640We present Voice2Face: a Deep Learning model that generates face and tongue animations directly from recorded speech. Our approach consists of two steps: a conditional Variational Autoencoder generates mesh animations from speech, while a separate module maps the animations to rig controller space. Our contributions include an automated method for speech style control, a method to train a model with data from multiple quality levels, and a method for animating the tongue. Unlike previous works, our model generates animations without speaker-dependent characteristics while allowing speech style control. We demonstrate through a user study that Voice2Face significantly outperforms a comparative state-of-the-art model in terms of perceived animation quality, and our quantitative evaluation suggests that Voice2Face yields more accurate lip closure in speech with bilabials through our speech style optimization. Both evaluations also show that our data quality conditioning scheme outperforms both an unconditioned model and a model trained with a smaller high-quality dataset. Finally, the user study shows a preference for animations including tongue. Results from our model can be seen at https://go.ea.com/voice2face.CCS Concepts: Computing methodologies --> Animation; Neural networks; Latent variable models; Learning latent representations; Additional Key Words and Phrases: Deep Learning, Facial animation, Tongue animation, Lip synchronization, Rig animationComputing methodologiesAnimationNeural networksLatent variable modelsLearning latent representationsAdditional Key Words and PhrasesDeep LearningFacial animationTongue animationLip synchronizationRig animation10.1111/cgf.14640255-26511 pages