* We recommend using the Chrome or Safari browser to correctly dispay all visuals.
We can render the same scene from continuous camera views. Unlike other methods, we ensure multiview consistency and do not "hide" wrinkles and shadows beneath the skin surface. Nerface suffers from high instability and often diverges, as seen in the second subject
We can also decompose each video into albedo, diffuse shading and specular shading.
Our method can control eye gaze and synthesize novel view directions. Note how our method is significantly sharper and moves smoother than all other competing methods. Mixture of Volumetric Primitives(MVP) requires a projected texture as an input, which is not given for such manipulations. We therefore use a neutral texture as a placeholder. MVP is unable to regaze properly, instead blending from one eye pose to the next. On the other hand, the lack of overall quality prevents the eyeball in EyeNeRF from being learned correctly for one pose of the first subject, and generally for the second subject.
We can do the same with a moving camera.
We interpolate between 13 expressions. Related works struggle to render convincing expressions. For our method, the periocular region smoothly adapts and shows detailed, naturally deforming wrinkles and highly detailed reflections on the eyeball. Again, for MVP we use the neutral texture as a placeholder.
We can do also perform interpolation of expressions while moving the camera and maintain 3D consistency throughout the motion. SotA methods struggle with gaze and expression changes and produce significant floaters for novel camera viewpoints.
Our shell-based formulation enables fine-grained control via a 3DMM parameters. We show slow-motion renderings of a highly complex expression: a closing eyelid. Only our method and MonoAvatar are capable of this.
We show results on expressions unseen during training. In these examples, we extract 3DMM coefficients from the Expression Target (left) and apply them to our target subject on the right. Note how our method faithfully applies the desired expressions. Note that our 3DMM fitting method does not enforce temporal smoothness, resulting in jitter. As we crop our image based on the estimated eye pose, this results in jitter in the GT video as well. Although performs comparably in most situations, it is unable to handle certain expressions which it did not directly see, such as the half open eye in the first expression, resulting in strong artifacting.
We show results on expressions unseen during training and interpolated expressions with camera orbits for all our ablation settings.