Explaining Why: How Instructions and User Interfaces Impact Annotator Rationales When Labeling Text Data
Explaining Why: How Instructions and User Interfaces Impact Annotator Rationales When Labeling Text Data
复制标题
DOI：
10.18653/v1/2022.naacl-main.38
复制
发表时间：
2022
影响因子：
16.6
通讯作者：
Jamar L. Sullivan;Will Brackenbury;Andrew McNut;K. Bryson;Kwam Byll;Yuxin Chen;M. Littman;Chenhao Tan;Blase Ur
中科院分区：
文献类型：
--
作者：
Jamar L. Sullivan;Will Brackenbury;Andrew McNut;K. Bryson;Kwam Byll;Yuxin Chen;M. Littman;Chenhao Tan;Blase Ur
在数据标签的背景下，NLP研究人员越来越感兴趣的是让人类选择合理性，这是与所选标签相关的输入的子集。我们进行了一项332名参与者的在线用户研究，以了解人类如何选择理由，特别是不同的说明和用户界面启示如何影响所选择的理由。参与者将十篇电影评论标记为积极或消极，选择支持他们标签的单词和短语作为理由。我们改变了给出的指令，合理选择任务和用户界面。参与者通常会选择大约12%的输入作为基本原理，但如果不能一次拖动多个标记，则选择较少。虽然参与者在数据标签上几乎一致，但他们在理论基础上却不那么一致。用户界面启示和任务极大地影响了选择的基本原理类型。我们还观察到参与者之间存在很大的差异。
In the context of data labeling, NLP researchers are increasingly interested in having humans select rationales , a subset of input to-kens relevant to the chosen label. We conducted a 332-participant online user study to understand how humans select rationales, especially how different instructions and user interface affordances impact the rationales chosen. Participants labeled ten movie reviews as positive or negative, selecting words and phrases supporting their label as rationales. We varied the instructions given, the rationale-selection task, and the user interface. Participants often selected about 12% of input to-kens as rationales, but selected fewer if unable to drag over multiple tokens at once. Whereas participants were near unanimous in their data labels, they were far less consistent in their rationales. The user interface affor-dances and task greatly impacted the types of rationales chosen. We also observed large variance across participants.