Fallacies of Agreement
Fallacies of Agreement
复制标题
协议的谬误
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Theophanis Tsandilas
中科院分区：
文献类型：
--
作者：
Theophanis Tsandilas
发现能获得共识的手势是手势引出的一个关键目标。为此，人机界面研究开发了统计方法来推理一致性。我们回顾了这些方法，并指出了三个主要问题。首先，我们发现原始协议比率忽略了偶然发生的协议，并且没有可靠地捕捉到参与者如何区分参照物。其次，我们解释了为什么当前关于如何解释协议分数的建议依赖于有问题的假设。第三，我们证明了用于比较协议比率的显著性检验，无论是在参与者内部还是在参与者之间，都会产生很大的I型错误率(α=0.05时为40%)。作为替代，我们提出了在评分者之间的可靠性研究中经常使用的一致性指数。我们讨论了如何将它们应用于手势引诱研究。我们还演示了如何使用常见的重采样技术来支持区间估计的统计推断。我们运用这些方法重新分析和解释了四项手势诱导研究的结果。
Discovering gestures that gain consensus is a key goal of gesture elicitation. To this end, HCI research has developed statistical methods to reason about agreement. We review these methods and identify three major problems. First, we show that raw agreement rates disregard agreement that occurs by chance and do not reliably capture how participants distinguish among referents. Second, we explain why current recommendations on how to interpret agreement scores rely on problematic assumptions. Third, we demonstrate that significance tests for comparing agreement rates, either within or between participants, yield large Type I error rates (>40% for α =.05). As alternatives, we present agreement indices that are routinely used in inter-rater reliability studies. We discuss how to apply them to gesture elicitation studies. We also demonstrate how to use common resampling techniques to support statistical inference with interval estimates. We apply these methods to reanalyze and reinterpret the findings of four gesture elicitation studies.