Collaborative Research: User-Centered Visual Analytics Evaluation

摘要：本提案的重点是扩展和编纂视觉分析评估的方法和技术。它主要基于2010年可视化分析挑战(由IEEE可视化分析科学与技术？作为一个测试平台，推广以用户为中心的评估，以评估结合分析推理、视觉表示、人机交互（HCI）、复杂算法和协作工具的交互系统的有效性。目前，还没有标准的方法来衡量交互式可视化分析系统作为一个整体的性能。视觉分析系统不太适合传统的评估方法。例如，简单的方法，如精度和召回率是不合适的，研究人员很少有足够的访问历史数据集的分析问题的复杂性，存在于更广泛的，实时的领域。执行该项目的方法包括三项活动：1)收集各种数据集，并对其进行相关的地面真相/问题描述；2)制定自动度量标准和主观评估标准，以评估判断；3)调查这些方法的有效性和效用，以及在商业、情报、医学、应急响应等其他动态领域中重用所获得的经验教训。

Abstract: This proposal focuses on extending and codifying the methods and techniques for the evaluation of visual analytics. It is primarily based on benchmarking the Visual Analytics Challenge 2010 (organized through the IEEE Visual Analytics Science and Technology ? known as VAST - symposium contest/challenge) as a test bed to generalize user-centered evaluation to assess the effectiveness of interactive systems that combine analytical reasoning, visual representations, Human Computer Interactions (HCI), complex algorithms, and collaboration tools. At present, there are no standard methods for measuring the performance of interactive visual analytics systems as a whole. Visual analysis systems are not well suited for traditional evaluation methods. For example, simple methods such as precision and recall are not suitable and researchers rarely have had sufficient access to historical data sets with the complexity of the analytical problems that exist across wider, real-time domains. The means of approach to executing this project encompasses three activities: 1) gathering diverse collation of datasets with associated ground truth/problem descriptions, 2) developing automatic metrics and subjective evaluation criteria to assess judgment, and 3) the investigation of the effectiveness and utility of these methods and lessons learned for re-use in other dynamic fields such as business, intelligence, medicine, emergency response, etc.