The Value of Activity Traces in Peer Evaluations: An Experimental Study
The Value of Activity Traces in Peer Evaluations: An Experimental Study
复制标题
DOI：
10.1145/3579627
复制
发表时间：
2023-04
影响因子：
--
通讯作者：
W. Shi;Sneha R. Krishna Kumaran;Hari Sundaram;B. Bailey
中科院分区：
文献类型：
--
作者：
W. Shi;Sneha R. Krishna Kumaran;Hari Sundaram;B. Bailey
同伴评价是在协作环境中评估个人和团队绩效的一种行之有效的工具，但容易受到社会和认知偏见的影响。目前的同行评估工具还没有解决在线协作技术为解决这些偏见提供的独特机会。在这项工作中，我们探索了同行评估的一个这样的机会的潜力：由协作工具自动生成的数据跟踪，我们称之为“活动跟踪”。我们对101名学生和MTurk员工进行了一项受试者间实验，调查在线协作任务中回顾活动轨迹对团队成员同伴评价的影响。我们的研究结果表明，与控制条件相比，使用活动痕迹导致参与者对他们的评估做出更多、更大的修改。这些修改也增加了一致性和参与者对他们收到的评估的感知准确性。我们的研究结果证明了活动轨迹作为一种对团队合作进行更可靠和客观的同行评估的方法的价值。基于我们的发现以及我们研究中自由形式回应的定性分析，我们还确定并讨论了将活动痕迹纳入现实世界同行评估系统的关键考虑因素和设计建议。
Peer evaluations are a well-established tool for evaluating individual and team performance in collaborative contexts, but are susceptible to social and cognitive biases. Current peer evaluation tools have also yet to address the unique opportunities that online collaborative technologies provide for addressing these biases. In this work, we explore the potential of one such opportunity for peer evaluations: data traces automatically generated by collaborative tools, which we refer to as "activity traces". We conduct a between-subjects experiment with 101 students and MTurk workers, investigating the effects of reviewing activity traces on peer evaluations of team members in an online collaborative task. Our findings show that the usage of activity traces led participants to make more and greater revisions to their evaluations compared to a control condition. These revisions also increased the consistency and participants' perceived accuracy of the evaluations that they received. Our findings demonstrate the value of activity traces as an approach for performing more reliable and objective peer evaluations of teamwork. Based on our findings as well as qualitative analysis of free-form responses in our study, we also identify and discuss key considerations and design recommendations for incorporating activity traces into real-world peer evaluation systems.