Collaborative Research: User-Centered Visual Analytics Evaluation

摘要：这项建议的重点是扩展和编纂视觉分析评估的方法和技术。它主要基于2010年视觉分析挑战赛的基准(通过IEEE视觉分析科学与技术组织？作为一个测试平台，推广以用户为中心的评估，以评估结合了分析推理、可视化表示、人机交互(HCI)、复杂算法和协作工具的交互系统的有效性。目前，还没有标准的方法来衡量交互式视觉分析系统的整体性能。可视化分析系统不能很好地适应传统的评估方法。例如，精确度和查全率等简单方法并不适用，研究人员很少有足够的机会访问历史数据集，这些数据集具有跨更广泛的实时领域存在的分析问题的复杂性。执行该项目的方法包括三项活动：1)收集具有相关基本事实/问题描述的数据集的不同校对；2)制定自动衡量标准和主观评估标准以评估判断；3)调查这些方法和经验教训的有效性和实用性，以便在其他动态领域，如商业、情报、医学、应急响应等中重复使用。

Abstract: This proposal focuses on extending and codifying the methods and techniques for the evaluation of visual analytics. It is primarily based on benchmarking the Visual Analytics Challenge 2010 (organized through the IEEE Visual Analytics Science and Technology ? known as VAST - symposium contest/challenge) as a test bed to generalize user-centered evaluation to assess the effectiveness of interactive systems that combine analytical reasoning, visual representations, Human Computer Interactions (HCI), complex algorithms, and collaboration tools. At present, there are no standard methods for measuring the performance of interactive visual analytics systems as a whole. Visual analysis systems are not well suited for traditional evaluation methods. For example, simple methods such as precision and recall are not suitable and researchers rarely have had sufficient access to historical data sets with the complexity of the analytical problems that exist across wider, real-time domains. The means of approach to executing this project encompasses three activities: 1) gathering diverse collation of datasets with associated ground truth/problem descriptions, 2) developing automatic metrics and subjective evaluation criteria to assess judgment, and 3) the investigation of the effectiveness and utility of these methods and lessons learned for re-use in other dynamic fields such as business, intelligence, medicine, emergency response, etc.