41-Issue 8

Permanent URI for this collection

https://diglib.eg.org/handle/10.2312/2633208

Browse

Now showing 1 - 20 of 31

Facial Animation with Disentangled Identity and Motion using Transformers
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Chandran, Prashanth; Zoss, Gaspard; Gross, Markus; Gotardo, Paulo; Bradley, Derek; Dominik L. Michels; Soeren Pirk
We propose a 3D+time framework for modeling dynamic sequences of 3D facial shapes, representing realistic non-rigid motion during a performance. Our work extends neural 3D morphable models by learning a motion manifold using a transformer architecture. More specifically, we derive a novel transformer-based autoencoder that can model and synthesize 3D geometry sequences of arbitrary length. This transformer naturally determines frame-to-frame correlations required to represent the motion manifold, via the internal self-attention mechanism. Furthermore, our method disentangles the constant facial identity from the time-varying facial expressions in a performance, using two separate codes to represent neutral identity and the performance itself within separate latent subspaces. Thus, the model represents identity-agnostic performances that can be paired with an arbitrary new identity code and fed through our new identity-modulated performance decoder; the result is a sequence of 3D meshes for the performance with the desired identity and temporal length. We demonstrate how our disentangled motion model has natural applications in performance synthesis, performance retargeting, key-frame interpolation and completion of missing data, performance denoising and retiming, and other potential applications that include full 3D body modeling.
Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Users
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Ye, Yongjing; Liu, Libin; Hu, Lei; Xia, Shihong; Dominik L. Michels; Soeren Pirk
Animating an avatar that reflects a user's action in the VR world enables natural interactions with the virtual environment. It has the potential to allow remote users to communicate and collaborate in a way as if they met in person. However, a typical VR system provides only a very sparse set of up to three positional sensors, including a head-mounted display (HMD) and optionally two hand-held controllers, making the estimation of the user's full-body movement a difficult problem. In this work, we present a data-driven physics-based method for predicting the realistic full-body movement of the user according to the transformations of these VR trackers and simulating an avatar character to mimic such user actions in the virtual world in realtime. We train our system using reinforcement learning with carefully designed pretraining processes to ensure the success of the training and the quality of the simulation. We demonstrate the effectiveness of the method with an extensive set of examples.
A Second Order Cone Programming Approach for Simulating Biphasic Materials
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Tang, Pengbin; Coros, Stelian; Thomaszewski, Bernhard; Dominik L. Michels; Soeren Pirk
Strain limiting is a widely used approach for simulating biphasic materials such as woven textiles and biological tissue that exhibit a soft elastic regime followed by a hard deformation limit. However, existing methods are either based on slowly converging local iterations, or offer no guarantees on convergence. In this work, we propose a new approach to strain limiting based on second order cone programming (SOCP). Our work is based on the key insight that upper bounds on per-triangle deformations lead to convex quadratic inequality constraints. Though nonlinear, these constraints can be reformulated as inclusion conditions on convex sets, leading to a second order cone programming problem-a convex optimization problem that a) is guaranteed to have a unique solution and b) allows us to leverage efficient conic programming solvers. We first cast strain limiting with anisotropic bounds on stretching as a quadratically constrained quadratic program (QCQP), then show how this QCQP can be mapped to a second order cone programming problem. We further propose a constraint reflection scheme and empirically show that it exhibits superior energy-preservation properties compared to conventional end-of-step projection methods. Finally, we demonstrate our prototype implementation on a set of examples and illustrate how different deformation limits can be used to model a wide range of material behaviors.
Sketching Vocabulary for Crowd Motion
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Mathew, C. D. Tharindu; Benes, Bedrich; Aliaga, Daniel; Dominik L. Michels; Soeren Pirk
This paper proposes and evaluates a sketching language to author crowd motion. It focuses on the path, speed, thickness, and density parameters of crowd motion. A sketch-based vocabulary is proposed for each parameter and evaluated in a user study against complex crowd scenes. A sketch recognition pipeline converts the sketches into a crowd simulation. The user study results show that 1) participants at various skill levels and can draw accurate crowd motion through sketching, 2) certain sketch styles lead to a more accurate representation of crowd parameters, and 3) sketching allows to produce complex crowd motions in a few seconds. The results show that some styles although accurate actually are less preferred over less accurate ones.
Pose Representations for Deep Skeletal Animation
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Andreou, Nefeli; Aristidou, Andreas; Chrysanthou, Yiorgos; Dominik L. Michels; Soeren Pirk
Data-driven skeletal animation relies on the existence of a suitable learning scheme, which can capture the rich context of motion. However, commonly used motion representations often fail to accurately encode the full articulation of motion, or present artifacts. In this work, we address the fundamental problem of finding a robust pose representation for motion, suitable for deep skeletal animation, one that can better constrain poses and faithfully capture nuances correlated with skeletal characteristics. Our representation is based on dual quaternions, the mathematical abstractions with well-defined operations, which simultaneously encode rotational and positional orientation, enabling a rich encoding, centered around the root. We demonstrate that our representation overcomes common motion artifacts, and assess its performance compared to other popular representations. We conduct an ablation study to evaluate the impact of various losses that can be incorporated during learning. Leveraging the fact that our representation implicitly encodes skeletal motion attributes, we train a network on a dataset comprising of skeletons with different proportions, without the need to retarget them first to a universal skeleton, which causes subtle motion elements to be missed. Qualitative results demonstrate the usefulness of the parameterization in skeleton-specific synthesis.
SCA 2022 CGF 41-8: Frontmatter
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Dominik L. Michels; Soeren Pirk; Dominik L. Michels; Soeren Pirk
Physically Based Shape Matching
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Müller, Matthias; Macklin, Miles; Chentanez, Nuttapong; Jeschke, Stefan; Dominik L. Michels; Soeren Pirk
The shape matching method is a popular approach to simulate deformable objects in interactive applications due to its stability and simplicity. An important feature is that there is no need for a mesh since the method works on arbitrary local groups within a set of particles. A major drawback of shape matching is the fact that it is geometrically motivated and not derived from physical principles which makes calibration difficult. The fact that the method does not conserve volume can yield visual artifacts, e.g. when a tire is compressed but does not bulge. In this paper we present a new meshless simulation method that is related to shape matching but derived from continuous constitutive models. Volume conservation and stiffness can be specified with physical parameters. Further, if the elements of a tetrahedral mesh are used as groups, our method perfectly reproduces FEM based simulations.
Learning Physics with a Hierarchical Graph Network
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Chentanez, Nuttapong; Jeschke, Stefan; Müller, Matthias; Macklin, Miles; Dominik L. Michels; Soeren Pirk
We propose a hierarchical graph for learning physics and a novel way to handle obstacles. The finest level of the graph consist of the particles itself. Coarser levels consist of the cells of sparse grids with successively doubling cell sizes covering the volume occupied by the particles. The hierarchical structure allows for the information to propagate at great distance in a single message passing iteration. The novel obstacle handling allows the simulation to be obstacle aware without the need for ghost particles. We train the network to predict effective acceleration produced by multiple sub-steps of 3D multi-material material point method (MPM) simulation consisting of water, sand and snow with complex obstacles. Our network produces lower error, trains up to 7.0X faster and inferences up to 11.3X faster than [SGGP*20]. It is also, on average, about 3.7X faster compared to Taichi Elements simulation running on the same hardware in our tests.
A Fusion Crowd Simulation Method: Integrating Data with Dynamics, Personality with Common
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Mao, Tianlu; Wang, Ji; Meng, Ruoyu; Yan, Qinyuan; Liu, Shaohua; Wang, Zhaoqi; Dominik L. Michels; Soeren Pirk
This paper proposes a novel crowd simulation method which integrates not only modelling ideas but also advantages from both data-driven methods and crowd dynamics methods. To seamlessly integrate these two different modelling ideas, first, a fusion crowd motion model is developed. In this model the motion of crowd are driven dynamically by different forces. Part of the forces are modeled under a universal interaction mechanism, which describe the common parts of crowd dynamics. Others are modeled by examples from real data, which describe the personality parts of the agent motion. Second, a construction method for example dataset is proposed to support the fusion model. In the dataset, crowd trajectories captured in the real world are decomposed and re-described under the structure of the fusion model. Thus, personality parts hidden in the real data could be locked and extracted, making the data understandable and migratable for our fusion model. A comprehensive crowd motion generation workflow using the fusion model and example dataset is also proposed. Quantitative and qualitative experiments and user studies are conducted. Results show that the proposed fusion crowd simulation method can generate crowd motion with the great motion fidelity, which not only match the macro characteristics of real data, but also has lots of micro personality showing the diversity of crowd motion.
Surface-Only Dynamic Deformables using a Boundary Element Method
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Sugimoto, Ryusuke; Batty, Christopher; Hachisuka, Toshiya; Dominik L. Michels; Soeren Pirk
We propose a novel surface-only method for simulating dynamic deformables without the need for volumetric meshing or volumetric integral evaluations. While based upon a boundary element method (BEM) for linear elastodynamics, our method goes beyond simple adoption of BEM by addressing several of its key limitations. We alleviate large displacement artifacts due to linear elasticity by extending BEM with a moving reference frame and surface-only fictitious forces, so that it only needs to handle deformations. To reduce memory and computational costs, we present a simple and practical method to compress the series of dense matrices required to simulate propagation of elastic waves over time. Furthermore, we explore a constraint enforcement mechanism and demonstrate the applicability of our method to general computer animation problems, such as frictional contact.
Wassersplines for Neural Vector Field-Controlled Animation
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Zhang, Paul; Smirnov, Dmitriy; Solomon, Justin; Dominik L. Michels; Soeren Pirk
Much of computer-generated animation is created by manipulating meshes with rigs. While this approach works well for animating articulated objects like animals, it has limited flexibility for animating less structured free-form objects. We introduce Wassersplines, a novel trajectory inference method for animating unstructured densities based on recent advances in continuous normalizing flows and optimal transport. The key idea is to train a neurally-parameterized velocity field that represents the motion between keyframes. Trajectories are then computed by advecting keyframes through the velocity field. We solve an additional Wasserstein barycenter interpolation problem to guarantee strict adherence to keyframes. Our tool can stylize trajectories through a variety of PDE-based regularizers to create different visual effects. We demonstrate our tool on various keyframe interpolation problems to produce temporally-coherent animations without meshing or rigging.
Differentiable Simulation for Outcome-Driven Orthognathic Surgery Planning
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Dorda, Daniel; Peter, Daniel; Borer, Dominik; Huber, Niko Benjamin; Sailer, Irena; Gross, Markus; Solenthaler, Barbara; Thomaszewski, Bernhard; Dominik L. Michels; Soeren Pirk
Algorithms at the intersection of computer graphics and medicine have recently gained renewed attention. A particular interest are methods for virtual surgery planning (VSP), where treatment parameters must be carefully chosen to achieve a desired treatment outcome. FEM simulators can verify the treatment parameters by comparing a predicted outcome to the desired one. However, estimating the optimal parameters amounts to solving a challenging inverse problem. In current clinical practice it is solved manually by surgeons, who rely on their experience and intuition to iteratively refine the parameters, verifying them with simulated predictions. We prototype a differentiable FEM simulator and explore how it can enhance and simplify treatment planning, which is ultimately necessary to integrate simulation-based VSP tools into a clinical workflow. Specifically, we define a parametric treatment model based on surgeon input, and with analytically derived simulation gradients we optimise it against an objective defined on the visible facial 3D surface. By using sensitivity analysis, we can easily explore the solution-space with first-order approximations, which allow the surgeon to interactively visualise the effect of parameter variations on a given treatment plan. The objective function allows landmarks to be freely chosen, accommodating the multiple methodologies in clinical planning. We show that even with a very sparse set of guiding landmarks, our simulator robustly converges to a feasible post-treatment shape.
MP-NeRF: Neural Radiance Fields for Dynamic Multi-person synthesis from Sparse Views
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Chao, Xian Jin; Leung, Howard; Dominik L. Michels; Soeren Pirk
Multi-person novel view synthesis aims to generate free-viewpoint videos for dynamic scenes of multiple persons. However, current methods require numerous views to reconstruct a dynamic person and only achieve good performance when only a single person is present in the video. This paper aims to reconstruct a multi-person scene with fewer views, especially addressing the occlusion and interaction problems that appear in the multi-person scene. We propose MP-NeRF, a practical method for multiperson novel view synthesis from sparse cameras without the pre-scanned template human models. We apply a multi-person SMPL template as the identity and human motion prior. Then we build a global latent code to integrate the relative observations among multiple people, so we could represent multiple dynamic people into multiple neural radiance representations from sparse views. Experiments on multi-person dataset MVMP show that our method is superior to other state-of-the-art methods.
Context-based Style Transfer of Tokenized Gestures
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Kuriyama, Shigeru; Mukai, Tomohiko; Taketomi, Takafumi; Mukasa, Tomoyuki; Dominik L. Michels; Soeren Pirk
Gestural animations in the amusement or entertainment field often require rich expressions; however, it is still challenging to synthesize characteristic gestures automatically. Although style transfer based on a neural network model is a potential solution, existing methods mainly focus on cyclic motions such as gaits and require re-training in adding new motion styles. Moreover, their per-pose transformation cannot consider the time-dependent features, and therefore motion styles of different periods and timings are difficult to be transferred. This limitation is fatal for the gestural motions requiring complicated time alignment due to the variety of exaggerated or intentionally performed behaviors. This study introduces a context-based style transfer of gestural motions with neural networks to ensure stable conversion even for exaggerated, dynamically complicated gestures. We present a model based on a vision transformer for transferring gestures' content and style features by time-segmenting them to compose tokens in a latent space. We extend this model to yield the probability of swapping gestures' tokens for style-transferring. A transformer model is suited to semantically consistent matching among gesture tokens, owing to the correlation with spoken words. The compact architecture of our network model requires only a small number of parameters and computational costs, which is suitable for real-time applications with an ordinary device. We introduce loss functions provided by the restoration error of identically and cyclically transferred gesture tokens and the similarity losses of content and style evaluated by splicing features inside the transformer. This design of losses allows unsupervised and zero-shot learning, by which the scalability for motion data is obtained. We comparatively evaluated our style transfer method, mainly focusing on expressive gestures using our dataset captured for various scenarios and styles by introducing new error metrics tailored for gestures. Our experiment showed the superiority of our method in numerical accuracy and stability of style transfer against the existing methods.
Stability Analysis of Explicit MPM
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Bai, Song; Schroeder, Craig; Dominik L. Michels; Soeren Pirk
In this paper we analyze the stability of the explicit material point method (MPM). We focus on PIC, APIC, and CPIC transfers using quadratic and cubic splines in two and three dimensions. We perform a fully three-dimensional Von Neumann stability analysis to study the behavior within the bulk of a material. This reveals the relationship between the sound speed, CFL number, and actual time step restriction and its dependence on discretization options. We note that boundaries are generally less stable than the interior, with stable time steps generally decreasing until the limit when particles become isolated. We then analyze the stability of a single particle to derive a novel time step restriction that stabilizes simulations at their boundaries. Finally, we show that for explicit MPM with APIC or CPIC transfers, there are pathological cases where growth is observed at arbitrarily small time steps sizes. While these cases do not necessarily pose a problem for practical usage, they do suggest that a guarantee of stability may be theoretically impossible and that necessary but not sufficient time step restrictions may be a necessary and practical compromise.
Monocular Facial Performance Capture Via Deep Expression Matching
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Bailey, Stephen W.; Riviere, Jérémy; Mikkelsen, Morten; O'Brien, James F.; Dominik L. Michels; Soeren Pirk
Facial performance capture is the process of automatically animating a digital face according to a captured performance of an actor. Recent developments in this area have focused on high-quality results using expensive head-scanning equipment and camera rigs. These methods produce impressive animations that accurately capture subtle details in an actor's performance. However, these methods are accessible only to content creators with relatively large budgets. Current methods using inexpensive recording equipment generally produce lower quality output that is unsuitable for many applications. In this paper, we present a facial performance capture method that does not require facial scans and instead animates an artist-created model using standard blendshapes. Furthermore, our method gives artists high-level control over animations through a workflow similar to existing commercial solutions. Given a recording, our approach matches keyframes of the video with corresponding expressions from an animated library of poses. A Gaussian process model then computes the full animation by interpolating from the set of matched keyframes. Our expression-matching method computes a low-dimensional latent code from an image that represents a facial expression while factoring out the facial identity. Images depicting similar facial expressions are identified by their proximity in the latent space. In our results, we demonstrate the fidelity of our expression-matching method. We also compare animations generated with our approach to animations generated with commercially available software.
A Second-Order Explicit Pressure Projection Method for Eulerian Fluid Simulation
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Jiang, Junwei; Shen, Xiangda; Gong, Yuning; Fan, Zeng; Liu, Yanli; Xing, Guanyu; Ren, Xiaohua; Zhang, Yanci; Dominik L. Michels; Soeren Pirk
In this paper, we propose a novel second-order explicit midpoint method to address the issue of energy loss and vorticity dissipation in Eulerian fluid simulation. The basic idea is to explicitly compute the pressure gradient at the middle time of each time step and apply it to the velocity field after advection. Theoretically, our solver can achieve higher accuracy than the first-order solvers at similar computational cost. On the other hand, our method is twice and even faster than the implicit second-order solvers at the cost of a small loss of accuracy. We have carried out a large number of 2D, 3D and numerical experiments to verify the effectiveness and availability of our algorithm.
Voice2Face: Audio-driven Facial and Tongue Rig Animations with cVAEs
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Villanueva Aylagas, Monica; Anadon Leon, Hector; Teye, Mattias; Tollmar, Konrad; Dominik L. Michels; Soeren Pirk
We present Voice2Face: a Deep Learning model that generates face and tongue animations directly from recorded speech. Our approach consists of two steps: a conditional Variational Autoencoder generates mesh animations from speech, while a separate module maps the animations to rig controller space. Our contributions include an automated method for speech style control, a method to train a model with data from multiple quality levels, and a method for animating the tongue. Unlike previous works, our model generates animations without speaker-dependent characteristics while allowing speech style control. We demonstrate through a user study that Voice2Face significantly outperforms a comparative state-of-the-art model in terms of perceived animation quality, and our quantitative evaluation suggests that Voice2Face yields more accurate lip closure in speech with bilabials through our speech style optimization. Both evaluations also show that our data quality conditioning scheme outperforms both an unconditioned model and a model trained with a smaller high-quality dataset. Finally, the user study shows a preference for animations including tongue. Results from our model can be seen at https://go.ea.com/voice2face.
Cognitive Model of Agent Exploration with Vision and Signage Understanding
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Johnson, Colin; Haworth, Brandon; Dominik L. Michels; Soeren Pirk
Signage systems play an essential role in ensuring safe, stress-free, and efficient navigation for the occupants of indoor spaces. Crowd simulations with sufficiently realistic virtual humans provide a convenient and cost-effective approach to evaluating and optimizing signage systems. In this work, we develop an agent model which makes use of image processing on parametric saliency maps to visually identify signage and distractions in the agent's field of view. Information from identified signs is incorporated into a grid-based representation of wayfinding familiarity, which is used to guide informed exploration of the agent's environment using a modified A* algorithm. In areas with low wayfinding familiarity, the agent follows a random exploration behaviour based on sampling a grid of previously observed locations for heuristic values based on space syntax isovist measures. The resulting agent design is evaluated in a variety of test environments and found to be able to reliably navigate towards a goal location using a combination of signage and random exploration.
Local Scale Adaptation to Hand Shape Model for Accurate and Robust Hand Tracking
(The Eurographics Association and John Wiley & Sons Ltd., 2022) Kalshetti, Pratik; Chaudhuri, Parag; Dominik L. Michels; Soeren Pirk
The accuracy of hand tracking algorithms depends on how closely the geometry of the mesh model resembles the user's hand shape. Most existing methods rely on a learned shape space model; however, this fails to generalize to unseen hand shapes with significant deviations from the training set. We introduce local scale adaptation to augment this data-driven shape model and thus enable modeling hands of substantially different sizes. We also present a framework to calibrate our proposed hand shape model by registering it to depth data and achieve accurate and robust tracking. We demonstrate the capability of our proposed adaptive shape model over the most widely used existing hand model by registering it to subjects from different demographics. We also validate the accuracy and robustness of our tracking framework on challenging public hand datasets where we improve over state-of-the-art methods. Our adaptive hand shape model and tracking framework offer a significant boost towards generalizing the accuracy of hand tracking.

Browse

Browsing 41-Issue 8 by Issue Date

Results Per Page

Sort Options