Can the online crowd match real expert judgments? How task complexity and coder location affect the validity of crowd‐coded data
Can the online crowd match real expert judgments? How task complexity and coder location affect the validity of crowd‐coded data
复制标题
在线人群可以匹配真实的专家判断吗？任务复杂性和编码器位置如何影响人群编码数据的有效性？
DOI：
--
复制
发表时间：
2019
影响因子：
5.3
通讯作者：
Alexander Horn
中科院分区：
文献类型：
--
作者：
Alexander Horn
Crowd-coding是一种新颖的技术，可以快速，负担得起和可重复的在线分类大量的报表。它结合了多个付费的非专家编码器的判断，以避免错误编码。有人认为，群众编码可以取代专家的判断，使用政治文本的编码作为一个例子，其中两种策略产生类似的结果。由于群体编码有可能将复制标准扩展到数据生产和基于少量精心设计的测试问题和答案的“规模”编码方案，因此更好地理解其可能性和局限性非常重要。虽然以前的低复杂度编码任务的结果是令人鼓舞的，这项研究评估是否以及在什么条件下简单和复杂的编码任务可以外包给人群，而不牺牲内容的有效性，以换取可扩展性。简单的任务是决定一个政党的声明是否算作对一个概念的积极提及--在这种情况下是平等。复杂的任务是区分五个平等概念。为了解释众包的上下文知识，IP限制是不同的。比较的基础是1 404份政党声明，由专家和群众编码（产生30 000份在线判决）。在陈述和政党宣言层面上对专家人群匹配的比较表明，即使对于复杂的任务，结果也基本相似，这表明复杂的类别方案可以通过人群编码来扩展。当IP限制被用作编码器专业知识的近似值时，匹配仅略高。
Crowd‐coding is a novel technique that allows for fast, affordable and reproducible online categorisation of large numbers of statements. It combines judgements by multiple, paid, non‐expert coders to avoid miscoding(s). It has been argued that crowd‐coding could replace expert judgements, using the coding of political texts as an example in which both strategies produce similar results. Since crowd‐coding yields the potential to extend the replication standard to data production and to ‘scale’ coding schemes based on a modest number of carefully devised test questions and answers, it is important that its possibilities and limitations are better understood. While previous results for low complexity coding tasks are encouraging, this study assesses whether and under what conditions simple and complex coding tasks can be outsourced to the crowd without sacrificing content validity in return for scalability. The simple task is to decide whether a party statement counts as positive reference to a concept – in this case: equality. The complex task is to distinguish between five concepts of equality. To account for the crowd‐coder's contextual knowledge, the IP restrictions are varied. The basis for comparisons is 1,404 party statements, coded by experts and the crowd (resulting in 30,000 online judgements). Comparisons of the expert‐crowd match at the level of statements and party manifestos show that the results are substantively similar even for the complex task, suggesting that complex category schemes can be scaled via crowd‐coding. The match is only slightly higher when IP restrictions are used as an approximation of coder expertise.