Search Results

Now showing 1 - 10 of 43
  • Item
    Towards a Neural Graphics Pipeline for Controllable Image Generation
    (The Eurographics Association and John Wiley & Sons Ltd., 2021) Chen, Xuelin; Cohen-Or, Daniel; Chen, Baoquan; Mitra, Niloy J.; Mitra, Niloy and Viola, Ivan
    In this paper, we leverage advances in neural networks towards forming a neural rendering for controllable image generation, and thereby bypassing the need for detailed modeling in conventional graphics pipeline. To this end, we present Neural Graphics Pipeline (NGP), a hybrid generative model that brings together neural and traditional image formation models. NGP decomposes the image into a set of interpretable appearance feature maps, uncovering direct control handles for controllable image generation. To form an image, NGP generates coarse 3D models that are fed into neural rendering modules to produce view-specific interpretable 2D maps, which are then composited into the final output image using a traditional image formation model. Our approach offers control over image generation by providing direct handles controlling illumination and camera parameters, in addition to control over shape and appearance variations. The key challenge is to learn these controls through unsupervised training that links generated coarse 3D models with unpaired real images via neural and traditional (e.g., Blinn- Phong) rendering functions, without establishing an explicit correspondence between them. We demonstrate the effectiveness of our approach on controllable image generation of single-object scenes. We evaluate our hybrid modeling framework, compare with neural-only generation methods (namely, DCGAN, LSGAN, WGAN-GP, VON, and SRNs), report improvement in FID scores against real images, and demonstrate that NGP supports direct controls common in traditional forward rendering. Code is available at http://geometry.cs.ucl.ac.uk/projects/2021/ngp.
  • Item
    High Dynamic Range Point Clouds for Real-Time Relighting
    (The Eurographics Association and John Wiley & Sons Ltd., 2019) Sabbadin, Manuele; Palma, Gianpaolo; BANTERLE, FRANCESCO; Boubekeur, Tamy; Cignoni, Paolo; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
    Acquired 3D point clouds make possible quick modeling of virtual scenes from the real world.With modern 3D capture pipelines, each point sample often comes with additional attributes such as normal vector and color response. Although rendering and processing such data has been extensively studied, little attention has been devoted using the light transport hidden in the recorded per-sample color response to relight virtual objects in visual effects (VFX) look-dev or augmented reality (AR) scenarios. Typically, standard relighting environment exploits global environment maps together with a collection of local light probes to reflect the light mood of the real scene on the virtual object. We propose instead a unified spatial approximation of the radiance and visibility relationships present in the scene, in the form of a colored point cloud. To do so, our method relies on two core components: High Dynamic Range (HDR) expansion and real-time Point-Based Global Illumination (PBGI). First, since an acquired color point cloud typically comes in Low Dynamic Range (LDR) format, we boost it using a single HDR photo exemplar of the captured scene that can cover part of it. We perform this expansion efficiently by first expanding the dynamic range of a set of renderings of the point cloud and then projecting these renderings on the original cloud. At this stage, we propagate the expansion to the regions not covered by the renderings or with low-quality dynamic range by solving a Poisson system. Then, at rendering time, we use the resulting HDR point cloud to relight virtual objects, providing a diffuse model of the indirect illumination propagated by the environment. To do so, we design a PBGI algorithm that exploits the GPU's geometry shader stage as well as a new mipmapping operator, tailored for G-buffers, to achieve real-time performances. As a result, our method can effectively relight virtual objects exhibiting diffuse and glossy physically-based materials in real time. Furthermore, it accounts for the spatial embedding of the object within the 3D environment. We evaluate our approach on manufactured scenes to assess the error introduced at every step from the perfect ground truth. We also report experiments with real captured data, covering a range of capture technologies, from active scanning to multiview stereo reconstruction.
  • Item
    Sequences with Low-Discrepancy Blue-Noise 2-D Projections
    (The Eurographics Association and John Wiley & Sons Ltd., 2018) Perrier, Hélène; Coeurjolly, David; Xie, Feng; Pharr, Matt; Hanrahan, Pat; Ostromoukhov, Victor; Gutierrez, Diego and Sheffer, Alla
    Distributions of samples play a very important role in rendering, affecting variance, bias and aliasing in Monte-Carlo and Quasi-Monte Carlo evaluation of the rendering equation. In this paper, we propose an original sampler which inherits many important features of classical low-discrepancy sequences (LDS): a high degree of uniformity of the achieved distribution of samples, computational efficiency and progressive sampling capability. At the same time, we purposely tailor our sampler in order to improve its spectral characteristics, which in turn play a crucial role in variance reduction, anti-aliasing and improving visual appearance of rendering. Our sampler can efficiently generate sequences of multidimensional points, whose power spectra approach so-called Blue-Noise (BN) spectral property while preserving low discrepancy (LD) in certain 2-D projections. In our tile-based approach, we perform permutations on subsets of the original Sobol LDS. In a large space of all possible permutations, we select those which better approach the target BN property, using pair-correlation statistics. We pre-calculate such ''good'' permutations for each possible Sobol pattern, and store them in a lookup table efficiently accessible in runtime. We provide a complete and rigorous proof that such permutations preserve dyadic partitioning and thus the LDS properties of the point set in 2-D projections. Our construction is computationally efficient, has a relatively low memory footprint and supports adaptive sampling. We validate our method by performing spectral/discrepancy/aliasing analysis of the achieved distributions, and provide variance analysis for several target integrands of theoretical and practical interest.
  • Item
    Moving Basis Decomposition for Precomputed Light Transport
    (The Eurographics Association and John Wiley & Sons Ltd., 2021) Silvennoinen, Ari; Sloan, Peter-Pike; Bousseau, Adrien and McGuire, Morgan
    We study the problem of efficient representation of potentially high-dimensional, spatially coherent signals in the context of precomputed light transport. We present a basis decomposition framework, Moving Basis Decomposition (MBD), that generalizes many existing basis expansion methods and enables high-performance, seamless reconstruction of compressed data. We develop an algorithm for solving large-scale MBD problems. We evaluate MBD against state-of-the-art in a series of controlled experiments and describe a real-world application, where MBD serves as the backbone of a scalable global illumination system powering multiple, current and upcoming 60Hz AAA-titles running on a wide range of hardware platforms.
  • Item
    Correlation-Aware Multiple Importance Sampling for Bidirectional Rendering Algorithms
    (The Eurographics Association and John Wiley & Sons Ltd., 2021) Grittmann, Pascal; Georgiev, Iliyan; Slusallek, Philipp; Mitra, Niloy and Viola, Ivan
    Combining diverse sampling techniques via multiple importance sampling (MIS) is key to achieving robustness in modern Monte Carlo light transport simulation. Many such methods additionally employ correlated path sampling to boost efficiency. Photon mapping, bidirectional path tracing, and path-reuse algorithms construct sets of paths that share a common prefix. This correlation is ignored by classical MIS heuristics, which can result in poor technique combination and noisy images.We propose a practical and robust solution to that problem. Our idea is to incorporate correlation knowledge into the balance heuristic, based on known path densities that are already required for MIS. This correlation-aware heuristic can achieve considerably lower error than the balance heuristic, while avoiding computational and memory overhead.
  • Item
    Stratified Sampling of Projected Spherical Caps
    (The Eurographics Association and John Wiley & Sons Ltd., 2018) Ureña, Carlos; Georgiev, Iliyan; Jakob, Wenzel and Hachisuka, Toshiya
    We present a method for uniformly sampling points inside the projection of a spherical cap onto a plane through the sphere's center. To achieve this, we devise two novel area-preserving mappings from the unit square to this projection, which is often an ellipse but generally has a more complex shape. Our maps allow for low-variance rendering of direct illumination from finite and infinite (e.g. sun-like) spherical light sources by sampling their projected solid angle in a stratified manner. We discuss the practical implementation of our maps and show significant quality improvement over traditional uniform spherical cap sampling in a production renderer.
  • Item
    Approximate Program Smoothing Using Mean-Variance Statistics, with Application to Procedural Shader Bandlimiting
    (The Eurographics Association and John Wiley & Sons Ltd., 2018) Yang, Yuting; Barnes, Connelly; Gutierrez, Diego and Sheffer, Alla
    We introduce a general method to approximate the convolution of a program with a Gaussian kernel. This results in the program being smoothed. Our compiler framework models intermediate values in the program as random variables, by using mean and variance statistics. We decompose the input program into atomic parts and relate the statistics of the different parts of the smoothed program. We give several approximate smoothing rules that can be used for the parts of the program. These include an improved variant of Dorn et al. [DBLW15], a novel adaptive Gaussian approximation, Monte Carlo sampling, and compactly supported kernels. Our adaptive Gaussian approximation handles multivariate Gaussian distributed inputs, gives exact results for a larger class of programs than previous work, and is accurate to the second order in the standard deviation of the kernel for programs with certain analytic properties. Because each expression in the program can have multiple approximation choices, we use a genetic search to automatically select the best approximations. We apply this framework to the problem of automatically bandlimiting procedural shader programs. We evaluate our method on a variety of geometries and complex shaders, including shaders with parallax mapping, animation, and spatially varying statistics. The resulting smoothed shader programs outperform previous approaches both numerically and aesthetically.
  • Item
    Temporally Reliable Motion Vectors for Real-time Ray Tracing
    (The Eurographics Association and John Wiley & Sons Ltd., 2021) Zeng, Zheng; Liu, Shiqiu; Yang, Jinglei; Wang, Lu; Yan, Ling-Qi; Mitra, Niloy and Viola, Ivan
    Real-time ray tracing (RTRT) is being pervasively applied. The key to RTRT is a reliable denoising scheme that reconstructs clean images from significantly undersampled noisy inputs, usually at 1 sample per pixel as limited by current hardware's computing power. The state of the art reconstruction methods all rely on temporal filtering to find correspondences of current pixels in the previous frame, described using per-pixel screen-space motion vectors. While these approaches are demonstrated powerful, they suffer from a common issue that the temporal information cannot be used when the motion vectors are not valid, i.e. when temporal correspondences are not obviously available or do not exist in theory. We introduce temporally reliable motion vectors that aim at deeper exploration of temporal coherence, especially for the generally-believed difficult applications on shadows, glossy reflections and occlusions, with the key idea to detect and track the cause of each effect. We show that our temporally reliable motion vectors produce significantly better temporal results on a variety of dynamic scenes when compared to the state of the art methods, but with negligible performance overhead.
  • Item
    An Efficient Transport Estimator for Complex Layered Materials
    (The Eurographics Association and John Wiley & Sons Ltd., 2020) Gamboa, Luis E.; Gruson, Adrien; Nowrouzezahrai, Derek; Panozzo, Daniele and Assarsson, Ulf
    Layered materials capture subtle, realistic reflection behaviors that traditional single-layer models lack. Much of this is due to the complex subsurface light transport at the interfaces of - and in the media between - layers. Rendering with these materials can be costly, since we must simulate these transport effects at every evaluation of the underlying reflectance model. Rendering an image requires thousands of such evaluations, per pixel. Recent work treats this complexity by introducing significant approximations, requiring large precomputed datasets per material, or simplifying the light transport simulations within the materials. Even the most effective of these methods struggle with the complexity induced by high-frequency variation in reflectance parameters and micro-surface normal variation, as well as anisotropic volumetric scattering between the layer interfaces. We present a more efficient, unbiased estimator for light transport in such general, complex layered appearance models. By conducting an analysis of the types of transport paths that contribute most to the aggregate reflectance dynamics, we propose an effective and unbiased path sampling method that reduces variance in the reflectance evaluations. Our method additionally supports reflectance importance sampling, does not rely on any precomputation, and so integrates readily into existing renderers. We consistently outperform the state-of-the-art by ~2-6x in equal-quality (i.e., equal error) comparisons.
  • Item
    Parallel Multiple-Bounce Irradiance Caching
    (The Eurographics Association and John Wiley & Sons Ltd., 2016) Jones, Nathaniel L.; Reinhart, Christoph F.; Elmar Eisemann and Eugene Fiume
    Building designers rely on predictive rendering techniques to design naturally and artificially lit environments. However, despite decades of work on the correctness of global illumination rendering techniques, our ability to accurately predict light levels in buildings-and to do so in a short time frame as part of an iterative design process-remains limited. In this paper, we present a novel approach to parallelizing construction of an irradiance cache over multiple-bounce paths. Relevant points for irradiance calculation based on one or multiple cameras are located by tracing rays through multiple-bounce paths. Irradiance values are then saved to a cache in reverse bounce order so that the irradiance calculation at each bounce samples from previously calculated values. We show by comparison to high-dynamic range photography of a moderately complex space that our method can predict luminance distribution as accurately as RADIANCE, the most widely validated tool used today for architectural predictive rendering of daylit spaces, and that it is faster by an order of magnitude.