Adler, A.Davis, R.Michiel van de Panne and Eric Saund2014-01-282014-01-282007978-3-905674-00-21812-3503https://doi.org/10.2312/SBM/SBM07/083-090Sketch recognition can capture the sketching component of a multimodal conversation about design, but it does not capture information conveyed in the other modalities. The informal speech that accompanies a sketch often has a considerable amount of additional information. We want to develop a digital whiteboard capable of understanding both sketching and speech, and capable of participating in a conversation similar to one that the user would have with a human design partner. We conducted a user study to help us understand what kinds of conversations users would have with a whiteboard capable of recognizing a sketch. We report results that we believe will help guide the design of an effective multimodal interface, and discuss implications for system architectures.Categories and Subject Descriptors (according to ACM CCS): H.5.2 [Information interfaces and presentation]: User Interfaces. - Natural language, Graphical user interfaces, Evaluation/methodology, Input devices and strategies, Interaction styles, User-centered design, Voice I/O