A Systematic Review on the Practice of Evaluating Visualization
A Systematic Review on the Practice of Evaluating Visualization
复制标题
DOI：
10.1109/tvcg.2013.126
复制
发表时间：
2013-12-01
影响因子：
5.2
通讯作者：
Moeller, Torsten
中科院分区：
文献类型：
--
作者：
Isenberg, Tobias;Isenberg, Petra;Moeller, Torsten
我们提出了一个评估的状态和历史发展的评估实践中发表的论文在IEEE可视化会议。我们的目标是通过系统地了解所提出的评价的特点和目标，在我们的社区中反映评价的元水平。为此，我们使用并扩展了Lam et al. [2012]先前建立的编码方案，对已发表论文中的10年评价进行了系统性综述。我们的审查结果包括社区中最常见的评估目标的概述，它们如何随着时间的推移而演变，以及它们如何与IEEE信息可视化会议的目标形成对比或一致。特别是，我们发现专门用于评估结果图像和算法性能的评估是最普遍的（自1997年以来，所有论文中始终有80-90%）。然而，特别是在过去六年中，包括参与者在内的评估方法稳步增加，要么通过评估他们的表现和主观反馈，要么通过评估他们的工作实践以及使用视觉工具改进的分析和推理能力。直到2010年，IEEE可视化会议的这一趋势比IEEE信息可视化会议更加明显，后者仅通过用户性能和体验测试显示出越来越多的评估百分比。然而，自2011年以来，IEEE信息可视化的论文也显示了使用可视化工具对工作实践和分析以及推理的评估的增加。此外，我们发现，一般的研究报告的需求分析和特定领域的工作实践过于非正式的报告，这阻碍了交叉比较，降低外部效度。
We present an assessment of the state and historic development of evaluation practices as reported in papers published at the IEEE Visualization conference. Our goal is to reflect on a meta-level about evaluation in our community through a systematic understanding of the characteristics and goals of presented evaluations. For this purpose we conducted a systematic review of ten years of evaluations in the published papers using and extending a coding scheme previously established by Lam et al. [2012]. The results of our review include an overview of the most common evaluation goals in the community, how they evolved over time, and how they contrast or align to those of the IEEE Information Visualization conference. In particular, we found that evaluations specific to assessing resulting images and algorithm performance are the most prevalent (with consistently 80-90% of all papers since 1997). However, especially over the last six years there is a steady increase in evaluation methods that include participants, either by evaluating their performances and subjective feedback or by evaluating their work practices and their improved analysis and reasoning capabilities using visual tools. Up to 2010, this trend in the IEEE Visualization conference was much more pronounced than in the IEEE Information Visualization conference which only showed an increasing percentage of evaluation through user performance and experience testing. Since 2011, however, also papers in IEEE Information Visualization show such an increase of evaluations of work practices and analysis as well as reasoning using visual tools. Further, we found that generally the studies reporting requirements analyses and domain-specific work practices are too informally reported which hinders cross-comparison and lowers external validity.