Li, GengyanSarkar, KripasindhuMeka, AbhimitraBuehler, MarcelMueller, FranziskaGotardo, PauloHilliges, OtmarBeeler, ThaboBermano, Amit H.Kalogerakis, Evangelos2024-04-302024-04-3020241467-8659https://doi.org/10.1111/cgf.15041https://diglib.eg.org/handle/10.1111/cgf15041Eye gaze and expressions are crucial non-verbal signals in face-to-face communication. Visual effects and telepresence demand significant improvements in personalized tracking, animation, and synthesis of the eye region to achieve true immersion. Morphable face models, in combination with coordinate-based neural volumetric representations, show promise in solving the difficult problem of reconstructing intricate geometry (eyelashes) and synthesizing photorealistic appearance variations (wrinkles and specularities) of eye performances. We propose a novel hybrid representation - ShellNeRF - that builds a discretized volume around a 3DMM face mesh using concentric surfaces to model the deformable 'periocular' region. We define a canonical space using the UV layout of the shells that constrains the space of dense correspondence search. Combined with an explicit eyeball mesh for modeling corneal light-transport, our model allows for animatable photorealistic 3D synthesis of the whole eye region. Using multi-view video input, we demonstrate significant improvements over state-of-the-art in expression re-enactment and transfer for high-resolution close-up views of the eye region.Attribution 4.0 International LicenseCCS Concepts: Computing methodologies -> Motion capture; Physical simulation; Image-based rendering; Mixed / augmented reality; Volumetric models; Parametric curve and surface models; Appearance and texture representations; Shape representations; 3D imaging; ReconstructionComputing methodologiesMotion capturePhysical simulationImagebased renderingMixed / augmented realityVolumetric modelsParametric curve and surface modelsAppearance and texture representationsShape representations3D imagingReconstruction10.1111/cgf.1504115 pages