III-CXT: Collaborative Research: Scientific Evaluation Methods for Visual Analytics Science and Technology

这是一项跨学科的机构间合作研究（0713087：Catherine Plaisant，马里兰州大学帕克分校; 0712770：Jean Scholtz，巴特尔纪念研究所; 0713198：乔治格林斯坦，马萨诸塞州洛厄尔大学），重点是视觉分析（VA），即，通过交互式视觉界面促进分析推理的科学。该项目解决了可视化分析方法和工具的一个重要方面，即开发一个评估基础设施，因为目前还没有就如何评估VA系统达成共识。评估其有效性尤其困难，因为它们将联合收割机的多个组成部分（分析推理、数据的视觉表示、计算机与人的互动、数据表示和算法、用于交流此类分析结果的工具）结合在一起，集成在复杂的系统中。此外，它是很难评估的有效性没有现实的数据和任务，因此，这是相当昂贵的每个研究人员来评估他们的具体VA方法的有效性。该项目的目标是设计和进行评估基础设施的初步测试，该基础设施将提供具有地面事实的数据集，为实验提供指导，测试方法和指标，并鼓励研究人员之间的协作和定性和定量结果的共享。由于视觉分析任务差异很大，从保持意识到评估情况，监控变化，解决犯罪或处理紧急情况，并且适用于具有不同需求的各种领域（例如，商业或情报分析、医学研究、应急管理），该项目旨在为这些不同的社区搭建桥梁。该项目的网站（http：//www.cs.umd.edu/hcil/semvast/）将包括一套可供分享的方法、工具和评价标准，以及该项目的其他成果。对视觉分析系统进行社区范围的系统评估，将更好地理解视觉分析所涉及的核心研究领域中的问题以及这些研究领域之间的交叉问题。所制定的评价方法将有利于研究活动和产品开发。这将带来更有效的系统，并影响所有可视化分析应用领域。视觉分析课程现在在大学和政府机构中教授。基准和自动评估工具将与教授和学生一起开发，并用于课堂项目和作业。

This is an interdisciplinary inter-institutional collaborative research (0713087: Catherine Plaisant, University of Maryland College Park; 0712770: Jean Scholtz, Battelle Memorial Institute; 0713198: George Grinstein, University of Massachusetts Lowell) focuses on visual analytics (VA), i.e., the science of analytical reasoning facilitated by interactive visual interfaces. This project addresses an important aspect of visual analytics methods and tools, namely developing an evaluation infrastructure, as there is currently no general consensus on how to evaluate VA systems. It is especially difficult to assess their effectiveness as they combine multiple components (analytical reasoning, visual representations of data, computer human interactions, data representations and algorithms, tools for communicating the results of such analyses) integrated in complex systems. Further, it is difficult to assess the effectiveness without realistic data and tasks; hence, it is quite costly for each individual researcher to evaluate the effectiveness of their specific VA approach. The goal of this project is to design and conduct initial tests of an evaluation infrastructure that will provide datasets with ground truth, supply guidance for experiments, test methodologies and metrics, and encourage collaboration and sharing of qualitative and quantitative results amongst researchers. Because visual analytics tasks vary widely, from maintaining awareness to assessing a situation, monitoring changes, solving crimes or dealing with emergencies, and are applicable to a variety of domains with different needs (e.g., business or intelligence analysis, medical research, emergency management), this project aims to bridge those diverse communities. The project Web site (http://www.cs.umd.edu/hcil/semvast/) will include a sharable set of methods, tools and metrics for evaluation, and other results from this project. Community wide, systematic evaluations of visual analytic systems will produce better understanding of the issues in the core research fields involved in visual analytics as well as the issues that cross between those research fields. The evaluation methodologies developed will benefit research activities as well as product development. This will lead to more effective systems and impact all visual analytics application domains. Classes in visual analysis are now being taught at the university level as well as in government agencies. Benchmarks and automated evaluation tools will be developed with professors and students and used in class projects and assignments.