Testing the reliability of inter-rater reliability
Testing the reliability of inter-rater reliability
复制标题
DOI：
10.1145/3375462.3375508
复制
发表时间：
2020-03
期刊：
影响因子：
--
通讯作者：
Brendan R. Eagan;Jais Brohinsky;Jingyi Wang;D. Shaffer
中科院分区：
文献类型：
--
作者：
Brendan R. Eagan;Jais Brohinsky;Jingyi Wang;D. Shaffer
对学习的分析通常依赖于编码数据。编码的一个重要方面是建立可靠性。先前的研究表明，用于建立编码可靠性的常见方法存在严重缺陷，因为它产生不可接受的高类型I错误率。本文的重点是测试这些错误率是否对应于特定的可靠性度量或更大的方法问题。我们的结果表明，建立可靠性的方法不是特定于度量的，我们建议采用新的实践来控制与建立编码可靠性相关的类型I错误率。
Analyses of learning often rely on coded data. One important aspect of coding is establishing reliability. Previous research has shown that the common approach for establishing coding reliability is seriously flawed in that it produces unacceptably high Type I error rates. This paper focuses on testing whether or not these error rates correspond to specific reliability metrics or a larger methodological problem. Our results show that the method for establishing reliability is not metric specific, and we suggest the adoption of new practices to control Type I error rates associated with establishing coding reliability.