III-CXT: Collaborative Research: Scientific Evaluation Methods for Visual Analytics Science and Technology

点击翻译按钮获取中文摘要

This is an interdisciplinary inter-institutional collaborative research (0713087: Catherine Plaisant, University of Maryland College Park; 0712770: Jean Scholtz, Battelle Memorial Institute; 0713198: George Grinstein, University of Massachusetts Lowell) focuses on visual analytics (VA), i.e., the science of analytical reasoning facilitated by interactive visual interfaces. This project addresses an important aspect of visual analytics methods and tools, namely developing an evaluation infrastructure, as there is currently no general consensus on how to evaluate VA systems. It is especially difficult to assess their effectiveness as they combine multiple components (analytical reasoning, visual representations of data, computer human interactions, data representations and algorithms, tools for communicating the results of such analyses) integrated in complex systems. Further, it is difficult to assess the effectiveness without realistic data and tasks; hence, it is quite costly for each individual researcher to evaluate the effectiveness of their specific VA approach. The goal of this project is to design and conduct initial tests of an evaluation infrastructure that will provide datasets with ground truth, supply guidance for experiments, test methodologies and metrics, and encourage collaboration and sharing of qualitative and quantitative results amongst researchers. Because visual analytics tasks vary widely, from maintaining awareness to assessing a situation, monitoring changes, solving crimes or dealing with emergencies, and are applicable to a variety of domains with different needs (e.g., business or intelligence analysis, medical research, emergency management), this project aims to bridge those diverse communities. The project Web site (http://www.cs.umd.edu/hcil/semvast/) will include a sharable set of methods, tools and metrics for evaluation, and other results from this project. Community wide, systematic evaluations of visual analytic systems will produce better understanding of the issues in the core research fields involved in visual analytics as well as the issues that cross between those research fields. The evaluation methodologies developed will benefit research activities as well as product development. This will lead to more effective systems and impact all visual analytics application domains. Classes in visual analysis are now being taught at the university level as well as in government agencies. Benchmarks and automated evaluation tools will be developed with professors and students and used in class projects and assignments.