Conversational Agent for Procedural Building Design in Virtual Reality
Loading...
Date
2024
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
With the emergence of large language models (LLMs), conversational agents have gained significant attention across various domains, including virtual reality (VR). This paper investigates the use of conversational agents as an interface for procedural building design in VR. We propose a voice interface that allows a user to control parameters of procedural generation and gain insights about the building construction metrics through natural conversation. The pipeline introduced for the conversational agent involves utilizing LLMs in two separate API calls for natural language understanding and natural language generation. This separation enables the invocation of various actions in procedural generation as well as meaningful agent responses to building-related questions. Furthermore, we conducted a user study to assess our proposed conversational interface in comparison to a traditional graphical user interface (GUI) in a VR architectural design task focused on circular economy. The study scrutinize the user-reported usability, presence, realism, errors, and effectiveness of both interfaces. Results suggest that while the non-embodied conversational agent enhances effectiveness due to its explanatory capabilities, it surprisingly decreases realism compared to the GUI. Overall, the preference between the conversational agent and the GUI varied greatly among participants, highlighting the need for further research into the evolving shift towards speech interaction in VR.
Description
CCS Concepts: Human-centered computing → Virtual reality; Natural language interfaces; Graphical user interfaces; Interactive systems and tools
@inproceedings{10.2312:egve.20241378,
booktitle = {ICAT-EGVE 2024 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments},
editor = {Hasegawa, Shoichi and Sakata, Nobuchika and Sundstedt, Veronica},
title = {{Conversational Agent for Procedural Building Design in Virtual Reality}},
author = {Bosco, Matteo and Kán, Peter and Kaufmann, Hannes},
year = {2024},
publisher = {The Eurographics Association},
ISSN = {1727-530X},
ISBN = {978-3-03868-245-5},
DOI = {10.2312/egve.20241378}
}