Controlled Crowdsourcing for High-Quality QA-SRL Annotation
Controlled Crowdsourcing for High-Quality QA-SRL Annotation
复制标题
DOI：
10.18653/v1/2020.acl-main.626
复制
发表时间：
2019-11
期刊：
影响因子：
--
通讯作者：
Paul Roit;Ayal Klein;Daniela Stepanov;Jonathan Mamou;Julian Michael;Gabriel Stanovsky;Luke Zettlemoyer;Ido Dagan
中科院分区：
文献类型：
--
作者：
Paul Roit;Ayal Klein;Daniela Stepanov;Jonathan Mamou;Julian Michael;Gabriel Stanovsky;Luke Zettlemoyer;Ido Dagan
问答驱动语义角色标注（问答驱动语义角色标注）是一种具有吸引力的、开放的、自然的语义角色标注，外行人也可以实现。最近，发布了一个大规模的众包QA-SRL语料库和一个经过训练的解析器。在尝试为新文本复制QA-SRL注释时，我们发现生成的注释质量较差，特别是在覆盖范围上，不足以进行进一步的研究和评估。在本文中，我们提出了一种改进的用于复杂语义注释的众包协议，包括工作人员的选择和培训，以及数据整合阶段。将该协议应用于QA-SRL产生了高质量的注释，覆盖率大大提高，产生了新的黄金评估数据集。我们相信我们的标注协议和金标准将促进未来自然语义标注的可复制研究。
Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially attainable from laymen. Recently, a large-scale crowdsourced QA-SRL corpus and a trained parser were released. Trying to replicate the QA-SRL annotation for new texts, we found that the resulting annotations were lacking in quality, particularly in coverage, making them insufficient for further research and evaluation. In this paper, we present an improved crowdsourcing protocol for complex semantic annotation, involving worker selection and training, and a data consolidation phase. Applying this protocol to QA-SRL yielded high-quality annotation with drastically higher coverage, producing a new gold evaluation dataset. We believe that our annotation protocol and gold standard will facilitate future replicable research of natural semantic annotations.