Evidence Humans Provide When Explaining Data-Labeling Decisions
Evidence Humans Provide When Explaining Data-Labeling Decisions
复制标题
DOI：
10.1007/978-3-030-29387-1_22
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
J. Newman;Bo Wang;Valerie Zhao;Amy Zeng;M. Littman;Blase Ur
中科院分区：
文献类型：
--
作者：
J. Newman;Bo Wang;Valerie Zhao;Amy Zeng;M. Littman;Blase Ur
由于机器学习将受益于减少的数据需求，一些先前的工作已经提出使用人类不仅要标记数据，还要解释这些标签。为了描述人类可能想要提供的证据，我们进行了用户研究和数据实验。在用户研究中，75名参与者为20张照片提供了分类标签，并通过自由文本解释来证明这些标签的合理性。解释经常引用图像中的概念（对象和属性），但26%的解释引用了图像中的概念。布尔逻辑是常见的隐式形式，但很少显式。在Visual Genome数据集的后续实验中，我们发现一些概念可以通过它们与频繁共现概念的关系来部分定义，而不仅仅是通过标记。
Because machine learning would benefit from reduced data requirements, some prior work has proposed using humans not just to label data, but also to explain those labels. To characterize the evidence humans might want to provide, we conducted a user study and a data experiment. In the user study, 75 participants provided classification labels for 20 photos, justifying those labels with free-text explanations. Explanations frequently referenced concepts (objects and attributes) in the image, yet 26% of explanations invoked conceptsnotin the image. Boolean logic was common in implicit form, but was rarely explicit. In a follow-up experiment on the Visual Genome dataset, we found that some concepts could be partially defined through their relationship to frequently co-occurring concepts, rather than only through labeling.