Scientific Evaluation in Visualization

Abstract

The objective of this half-day introductory tutorial is to increase awareness of what constitutes a sound scientific approach to evaluation in Visualization and to provide basic theoretical knowledge of and practical skills in current research practice. The content presents the current challenges and trends related to how to characterize and optimize the complex interactive visual displays present in Visualization today. It will cover the most basic and relevant issues to consider during different phases of evaluation: planning, design, execution, analysis of results and reporting. The content outlines how to proceed to achieve high quality results and point out common pitfalls and mistakes which are threats to high quality results. The main focus is on quantitative experimental research but the general knowledge applies to all kinds of studies. The tutorial will present the main part of the content by means of a lecture style using power-point presentations, and will use example studies from the tutorial leaders' own publications as well as other relevant work. There will also be demonstrations of different techniques for capturing data during an evaluation study. The participants will be given the opportunity to try out some of these methods hands-on to further facilitate a discussion of their potential suitability for different kinds of studies. Taking part in this tutorial will not train a novice participant to be fully capable of designing and conducting an evaluation study and analyzing its outcome, such a goal would require a substantially larger course. The aim is to introduce the topic, provide a general knowledge about what is important to consider and what resources are available to guide them in further study in this area. Further, participants will also learn to better judge the relevance and quality of a publication presenting an evaluation when reviewing such work since the same rules apply.