• Login
    View Item 
    •   Eurographics DL Home
    • Computer Graphics Forum
    • Volume 41 (2022)
    • 41-Issue 2
    • View Item
    •   Eurographics DL Home
    • Computer Graphics Forum
    • Volume 41 (2022)
    • 41-Issue 2
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers

    Thumbnail
    View/Open
    v41i2pp029-040.pdf (13.67Mb)
    hair_video_examples.mp4 (52.91Mb)
    lipstick_video_examples.mp4 (57.23Mb)
    Date
    2022
    Author
    Kips, Robin
    Jiang, Ruowei
    Ba, Sileye
    Duke, Brendan
    Perrot, Matthieu
    Gori, Pietro
    Bloch, Isabelle
    Pay-Per-View via TIB Hannover:

    Try if this item/paper is available.

    Metadata
    Show full item record
    Abstract
    Augmented reality applications have rapidly spread across online retail platforms and social media, allowing consumers to virtually try-on a large variety of products, such as makeup, hair dying, or shoes. However, parametrizing a renderer to synthesize realistic images of a given product remains a challenging task that requires expert knowledge. While recent work has introduced neural rendering methods for virtual try-on from example images, current approaches are based on large generative models that cannot be used in real-time on mobile devices. This calls for a hybrid method that combines the advantages of computer graphics and neural rendering approaches. In this paper, we propose a novel framework based on deep learning to build a real-time inverse graphics encoder that learns to map a single example image into the parameter space of a given augmented reality rendering engine. Our method leverages self-supervised learning and does not require labeled training data, which makes it extendable to many virtual try-on applications. Furthermore, most augmented reality renderers are not differentiable in practice due to algorithmic choices or implementation constraints to reach real-time on portable devices. To relax the need for a graphics-based differentiable renderer in inverse graphics problems, we introduce a trainable imitator module. Our imitator is a generative network that learns to accurately reproduce the behavior of a given non-differentiable renderer. We propose a novel rendering sensitivity loss to train the imitator, which ensures that the network learns an accurate and continuous representation for each rendering parameter. Automatically learning a differentiable renderer, as proposed here, could be beneficial for various inverse graphics tasks. Our framework enables novel applications where consumers can virtually try-on a novel unknown product from an inspirational reference image on social media. It can also be used by computer graphics artists to automatically create realistic rendering from a reference product image.
    BibTeX
    @article {10.1111:cgf.14456,
    journal = {Computer Graphics Forum},
    title = {{Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers}},
    author = {Kips, Robin and Jiang, Ruowei and Ba, Sileye and Duke, Brendan and Perrot, Matthieu and Gori, Pietro and Bloch, Isabelle},
    year = {2022},
    publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
    ISSN = {1467-8659},
    DOI = {10.1111/cgf.14456}
    }
    URI
    https://doi.org/10.1111/cgf.14456
    https://diglib.eg.org:443/handle/10.1111/cgf14456
    Collections
    • 41-Issue 2

    Related items

    Showing items related by title, author, creator and subject.

    • Rational Bézier Guarding 

      Khanteimouri, Payam; Mandad, Manish; Campen, Marcel (The Eurographics Association and John Wiley & Sons Ltd., 2022)
      We present a reliable method to generate planar meshes of nonlinear rational triangular elements. The elements are guaranteed to be valid, i.e. defined by injective rational functions. The mesh is guaranteed to conform ...
    • VA + Embeddings STAR: A State-of-the-Art Report on the Use of Embeddings in Visual Analytics 

      Huang, Zeyang; Witschard, Daniel; Kucher, Kostiantyn; Kerren, Andreas (The Eurographics Association and John Wiley & Sons Ltd., 2023)
      Over the past years, an increasing number of publications in information visualization, especially within the field of visual analytics, have mentioned the term ''embedding'' when describing the computational approach. ...
    • Personalised Authentic assessments with Synchronous Learning Activities: a Framework for Teaching Visualisation and Graphics 

      Roberts, Jonathan C. (The Eurographics Association, 2022)
      We present an activities framework for learning visualisation and computer graphics. The framework pivots around the academic developing an authentic learning scenario that is personalised for every student, followed by a ...

    Eurographics Association copyright © 2013 - 2023 
    Send Feedback | Contact - Imprint | Data Privacy Policy | Disable Google Analytics
    Theme by @mire NV
    System hosted at  Graz University of Technology.
    TUGFhA
     

     

    Browse

    All of Eurographics DLCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

    My Account

    LoginRegister

    Statistics

    View Usage Statistics

    BibTeX | TOC

    Create BibTeX Create Table of Contents

    Eurographics Association copyright © 2013 - 2023 
    Send Feedback | Contact - Imprint | Data Privacy Policy | Disable Google Analytics
    Theme by @mire NV
    System hosted at  Graz University of Technology.
    TUGFhA