Toward Three-Stage Automation of Annotation for Human Values
Toward Three-Stage Automation of Annotation for Human Values
复制标题
迈向人类价值注释的三阶段自动化
DOI：
10.1007/978-3-030-15742-5_18
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
An-Shou Cheng
中科院分区：
文献类型：
--
作者：
Emi Ishita;Satoshi Fukuda;Toru Oga;Douglas W. Oard;Kenneth R. Fleischmann;Yoichi Tomiura;An-Shou Cheng
先前关于人类价值观的自动注释的工作试图训练文本分类技术，以用反映特定人类价值观（如自由、正义或安全）的标签来标记文本跨度。这混淆了三项任务：（1）选择要标记的文档，（2）选择表达或反映人类价值的文本跨度，以及（3）为这些跨度分配标签。本文提出了一个三阶段模型，在该模型中，可以为三个阶段中的每个阶段优化训练单独的系统。实验从第一阶段，文件选择，注释多样性胜过注释质量，这表明，当多个注释器可用，传统的做法，裁定冲突的注释相同的文件是不符合成本效益的替代方案，其中每个注释器标签不同的文件。第二阶段的初步结果表明，选择有价值的句子可以在该任务上实现高召回率（94%），精确度（80%以上）似乎适合用作多阶段注释管道的一部分。为这些实验制作的注释免费提供，注释的内容可以从商业来源以适当的价格获得。
Prior work on automated annotation of human values has sought to train text classification techniques to label text spans with labels that reflect specific human values such as freedom, justice, or safety. This confounds three tasks: (1) selecting the documents to be labeled, (2) selecting the text spans that express or reflect human values, and (3) assigning labels to those spans. This paper proposes a three-stage model in which separate systems can be optimally trained for each of the three stages. Experiments from the first stage, document selection, indicate that annotation diversity trumps annotation quality, suggesting that when multiple annotators are available, the traditional practice of adjudicating conflicting annotations of the same documents is not as cost effective as an alternative in which each annotator labels different documents. Preliminary results for the second stage, selecting value sentences, indicate that high recall (94%) can be achieved on that task with levels of precision (above 80%) that seem suitable for use as part of a multi-stage annotation pipeline. The annotations created for these experiments are being made freely available, and the content that was annotated is available from commercial sources at modest cost.