ReflectanceFusion: Diffusion-based text to SVBRDF Generation

Simple item page
dc.contributor.authorXue, Bowenen_US
dc.contributor.authorGuarnera, Giuseppe Claudioen_US
dc.contributor.authorZhao, Shuangen_US
dc.contributor.authorMontazeri, Zahraen_US
dc.contributor.editorHaines, Ericen_US
dc.contributor.editorGarces, Elenaen_US
dc.date.accessioned2024-06-25T11:05:47Z
dc.date.available2024-06-25T11:05:47Z
dc.date.issued2024
dc.description.abstractWe introduce ReflectanceFusion (Reflectance Diffusion), a new neural text-to-texture model capable of generating high-fidelity SVBRDF maps from textual descriptions. Our method leverages a tandem neural approach, consisting of two modules, to accurately model the distribution of spatially varying reflectance as described by text prompts. Initially, we employ a pre-trained stable diffusion 2 model to generate a latent representation that informs the overall shape of the material and serves as our backbone model. Then, our ReflectanceUNet enables fine-tuning control over the material's physical appearance and generates SVBRDF maps. ReflectanceUNet module is trained on an extensive dataset comprising approximately 200,000 synthetic spatially varying materials. Our generative SVBRDF diffusion model allows for the synthesis of multiple SVBRDF estimates from a single textual input, offering users the possibility to choose the output that best aligns with their requirements. We illustrate our method's versatility by generating SVBRDF maps from a range of textual descriptions, both specific and broad. Our ReflectanceUNet model can integrate optional physical parameters, such as roughness and specularity, enhancing customization. When the backbone module is fixed, the ReflectanceUNet module refines the material, allowing direct edits to its physical attributes. Comparative evaluations demonstrate that ReflectanceFusion achieves better accuracy than existing text-to-material models, such as Text2Mat, while also providing the benefits of editable and relightable SVBRDF maps.en_US
dc.description.sectionheadersLight and Textures
dc.description.seriesinformationEurographics Symposium on Rendering
dc.identifier.doi10.2312/sr.20241152
dc.identifier.isbn978-3-03868-262-2
dc.identifier.issn1727-3463
dc.identifier.pages10 pages
dc.identifier.urihttps://doi.org/10.2312/sr.20241152
dc.identifier.urihttps://diglib.eg.org/handle/10.2312/sr20241152
dc.publisherThe Eurographics Associationen_US
dc.rightsAttribution 4.0 International License
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectCCS Concepts: Computing methodologies -> Reflectance modeling
dc.subjectCCS Concepts
dc.subjectComputing methodologies
dc.subject> Reflectance modeling
dc.titleReflectanceFusion: Diffusion-based text to SVBRDF Generationen_US
Files
Original bundle
Now showing 1 - 3 of 3
No Thumbnail Available
Name:
07_sr20241152.pdf
Size:
106.77 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
paper1045_supp.pdf
Size:
50.34 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
paper1045.mp4
Size:
112.03 MB
Format:
Video MP4
Download
Collections
Rendering 2024 - Symposium Track