III-CXT: Collaborative Research: Scientific Evaluation Methods for Visual Analytics Science and Technology

这是一项跨学科、跨机构的合作研究(0713087：凯瑟琳·普莱桑特，马里兰大学帕克分校；0712770：让·肖尔茨，巴特尔纪念研究所；0713198：乔治·格林斯坦，马萨诸塞大学洛厄尔分校)专注于视觉分析(VA)，即通过交互视觉界面促进分析推理的科学。该项目涉及视觉分析方法和工具的一个重要方面，即开发评价基础设施，因为目前没有就如何评价退伍军人制度达成普遍共识。评估它们的有效性尤其困难，因为它们结合了复杂系统中集成的多个组成部分(分析推理、数据的可视化表示、计算机与人的交互、数据表示和算法、用于传达此类分析结果的工具)。此外，如果没有现实的数据和任务，很难评估有效性；因此，每个研究人员评估其特定VA方法的有效性是相当昂贵的。该项目的目标是设计和进行评价基础设施的初步测试，该基础设施将提供具有基本事实的数据集，为实验、测试方法和衡量标准提供指导，并鼓励研究人员之间合作和分享定性和定量结果。由于视觉分析任务差异很大，从保持认识到评估情况、监测变化、解决犯罪或处理紧急情况，并适用于具有不同需求的各种领域(例如，商业或情报分析、医学研究、紧急情况管理)，因此该项目旨在将这些不同的社区联系起来。项目网站(http://www.cs.umd.edu/hcil/semvast/)将包括一套可共享的评估方法、工具和指标，以及来自该项目的其他结果。整个社区对视觉分析系统的系统评估将使人们更好地理解视觉分析涉及的核心研究领域中的问题，以及这些研究领域之间的交叉问题。制定的评价方法将有利于研究活动和产品开发。这将导致更有效的系统，并影响所有视觉分析应用领域。视觉分析课程现在正在大学和政府机构授课。将与教授和学生一起开发基准和自动化评估工具，并在课堂项目和作业中使用。

This is an interdisciplinary inter-institutional collaborative research (0713087: Catherine Plaisant, University of Maryland College Park; 0712770: Jean Scholtz, Battelle Memorial Institute; 0713198: George Grinstein, University of Massachusetts Lowell) focuses on visual analytics (VA), i.e., the science of analytical reasoning facilitated by interactive visual interfaces. This project addresses an important aspect of visual analytics methods and tools, namely developing an evaluation infrastructure, as there is currently no general consensus on how to evaluate VA systems. It is especially difficult to assess their effectiveness as they combine multiple components (analytical reasoning, visual representations of data, computer human interactions, data representations and algorithms, tools for communicating the results of such analyses) integrated in complex systems. Further, it is difficult to assess the effectiveness without realistic data and tasks; hence, it is quite costly for each individual researcher to evaluate the effectiveness of their specific VA approach. The goal of this project is to design and conduct initial tests of an evaluation infrastructure that will provide datasets with ground truth, supply guidance for experiments, test methodologies and metrics, and encourage collaboration and sharing of qualitative and quantitative results amongst researchers. Because visual analytics tasks vary widely, from maintaining awareness to assessing a situation, monitoring changes, solving crimes or dealing with emergencies, and are applicable to a variety of domains with different needs (e.g., business or intelligence analysis, medical research, emergency management), this project aims to bridge those diverse communities. The project Web site (http://www.cs.umd.edu/hcil/semvast/) will include a sharable set of methods, tools and metrics for evaluation, and other results from this project. Community wide, systematic evaluations of visual analytic systems will produce better understanding of the issues in the core research fields involved in visual analytics as well as the issues that cross between those research fields. The evaluation methodologies developed will benefit research activities as well as product development. This will lead to more effective systems and impact all visual analytics application domains. Classes in visual analysis are now being taught at the university level as well as in government agencies. Benchmarks and automated evaluation tools will be developed with professors and students and used in class projects and assignments.