# Learning To Stabilize Faces

J. Bednarik, E. Wood, V. Choutas, T. Bolkart, D. Wang, C. Wu, T. Beeler

Google

This supplementary page contains comparison of the compared methods OUR, CMAP, UNPOSE and PROCupper (see Section 4.3. of the main paper) on the task of stabilizing a facial performance. The input registered meshes were obtained by a custom fitting method (as described in Section 3.2. of the main paper) which, as opposed to fitting random sets of expressions, contain temporal smoothing terms which reduce both the amount of surface geometry noise and the spatial transformation jitter.

On the left of each video you can see the original frontal camera view from our multi-camera capture studio. Second to the left is our original unstabilized registered mesh and the remaining columns showcase the stabilized result produced by OUR and each of the compared method. The meshes are viewed slightly side-on to let the viewer better assess forwards-and-backwards motion as well as sideways motion. As you can see, our learning-based stabilization method is capable of removing rigid head motion from the facial performance.

Note: some performances exhibit artifacts (e.g. neck jitter or the surface geometry noise in the hair region) in the original registrations. Neither of the stabilization methods can remove such artifacts. Further: since we only rigidly stabilize the face, neck motion in the results is expected.

# Typical Failure Modes of the Compared Methods

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

This performance contains small but abrupt head rotations along the frontal axis, i.e. side-to-side head tilting along the coronal plane. Note, especially when the subject turns their lips downs and tilts the head at the same time, the method UNPOSE introduces a spurious global motion where the chin appears to draw inwards to compensate for the neck rotation.

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

The subject conitnuously moves and rotates their head as they are speaking, and this global motion is correctly undone by al the methods. However, note that when the subject blinks, the method CMAP subtly yet visibly rotates the head upwards which, when focusing on the texture, appears as if the subject raised their eyeborows or the forehead. However, it is evident from the input RGB video that no such motion happened in reality.

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Over the course of the utterance, the subject mildly but continuously rotates and moves their head. The performance represents a neutral speech and there are no extreme expressions, thus there should not appear any strong motion of the ears. However, by focusing on the right ear of the stabilized sequences, one can notice that when compared to the other methods, UNPOSE manifests much more visible global motion.

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Over the course of the utterance, the subject is quite abruptly rotating and moving their head. Note that as the head tilts side-to-side along the coronal plane, the method UNPOSE wrongly compensates by a global head rotation. During the whole performance, when focusing on the forehead region and the occlusion boundary of the head scalp, one can notice subtle but continuous spurious global motion introduced by the method CMAP.

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

The performance contains high-frequency low-anplitude global head motion. When focusing on the nose bridge area of the stabilized meshes, a small spurious leftover motion is visible in the UNPOSE method.

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Similarly to the video above, the nose bridge area of the stabilized meshes reveals a subtle unwanted motion left by the method UNPOSE.

# More Stabilization Results

We provide more videos comparing the competing methods on the stabilization of performances below.

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Input video

Original

OUR

CMAP

UNPOSE

PROCupper

Input video

Original

OUR

CMAP

UNPOSE

PROCupper