Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels.
Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels.
复制标题
通过众包软目标标签来训练情感计算机视觉模型。
DOI：
10.1007/s12559-021-09936-4
复制
发表时间：
2021-09
影响因子：
5.4
通讯作者：
Wall, Dennis P.
中科院分区：
文献类型：
--
作者：
Washington, Peter;Kalantarian, Haik;Kent, Jack;Husic, Arman;Kline, Aaron;Leblanc, Emilie;Hou, Cathy;Mutlu, Cezmi;Dunlap, Kaitlyn;Penev, Yordan;Stockham, Nate;Chrisman, Brianna;Paskov, Kelley;Jung, Jae-Yoon;Voss, Catalin;Haber, Nick;Wall, Dennis P.
情绪检测分类器通常预测离散的情绪。然而，情感表达往往是主观的，因此需要一种方法来处理复合和模糊的标签。我们探索了使用众包来获取可靠的软目标标签的可行性，并评估了用这些标签训练的情感检测分类器。我们假设，使用代表人类对图像解释多样性的标签进行训练，将导致在不相交测试集中具有类似代表性的预测。我们还假设，众包可以生成与实验室环境中生成的版本相对应的发行版。我们的研究集中在儿童情感面部表情（CAFE）数据集上，这是一个描绘儿童面部表情的黄金标准图像集合，每张图像有100个人类标签。为了测试众包生成这些标签的可行性，我们使用微工作者获取207张CAFE图像的标签。我们既评估了未经过滤的工人，也评估了通过短时间人群过滤过程选择的工人。然后，我们使用数据集提供的原始100个注释在软目标CAFE标签上训练两个版本的ResNet-152神经网络：(1)使用传统的单热编码标签训练的分类器，以及(2)使用表示CAFE注释器响应分布的向量标签训练的分类器。我们将得到的两个分类器的softmax输出分布与分类器输出概率分布与人类标签分布之间的L1距离的2样本独立t检验进行比较。虽然未经过滤的人群工作者对CAFE的认同程度较低，但过滤后的人群对快乐、中性、悲伤和“恐惧+惊讶”的CAFE标签的认同率为100%，对“愤怒+厌恶”的认同率为88.8%。虽然单热编码分类器的f1分数相对于真实CAFE标签要高得多（94.33% vs. 78.68%），但群体训练分类器的输出概率向量更接近于人类标签的分布（t=3.2827, p=0.0014）。对于情感计算的许多应用，报告情感概率分布可以解释人类解释的主观性，这可能比绝对标签更有用。众包是获得软目标标签的可行方案，其中包括一个充分的筛选机制来选择可靠的群体工作者。
Emotion detection classifiers traditionally predict discrete emotions. However, emotion expressions are often subjective, thus requiring a method to handle compound and ambiguous labels. We explore the feasibility of using crowdsourcing to acquire reliable soft-target labels and evaluate an emotion detection classifier trained with these labels. We hypothesize that training with labels that are representative of the diversity of human interpretation of an image will result in predictions that are similarly representative on a disjoint test set. We also hypothesize that crowdsourcing can generate distributions which mirror those generated in a lab setting. We center our study on the Child Affective Facial Expression (CAFE) dataset, a gold standard collection of images depicting pediatric facial expressions along with 100 human labels per image. To test the feasibility of crowdsourcing to generate these labels, we used Microworkers to acquire labels for 207 CAFE images. We evaluate both unfiltered workers as well as workers selected through a short crowd filtration process. We then train two versions of a ResNet-152 neural network on soft-target CAFE labels using the original 100 annotations provided with the dataset: (1) a classifier trained with traditional one-hot encoded labels, and (2) a classifier trained with vector labels representing the distribution of CAFE annotator responses. We compare the resulting softmax output distributions of the two classifiers with a 2-sample independent t-test of L1 distances between the classifier’s output probability distribution and the distribution of human labels. While agreement with CAFE is weak for unfiltered crowd workers, the filtered crowd agree with the CAFE labels 100% of the time for happy, neutral, sad and “fear + surprise”, and 88.8% for “anger + disgust”. While the F1-score for a one-hot encoded classifier is much higher (94.33% vs. 78.68%) with respect to the ground truth CAFE labels, the output probability vector of the crowd-trained classifier more closely resembles the distribution of human labels (t=3.2827, p=0.0014). For many applications of affective computing, reporting an emotion probability distribution that accounts for the subjectivity of human interpretation can be more useful than an absolute label. Crowdsourcing, including a sufficient filtering mechanism for selecting reliable crowd workers, is a feasible solution for acquiring soft-target labels.
登录
查看更多内容
影响因子：
3.9
作者：
Duda M;Daniels J;Wall DP
通讯作者：
Wall DP
影响因子：
4.2
作者：
Gray, Katie L. H.;Adams, Wendy J.;Garner, Matthew
通讯作者：
Garner, Matthew
DOI：
10.1073/pnas.1322355111
发表时间：
2014-04-15
影响因子：
11.1
作者：
Du, Shichuan;Tao, Yong;Martinez, Aleix M.
通讯作者：
Martinez, Aleix M.
影响因子：
15.2
作者：
Daniels, Jena;Schwartz, Jessey N.;Wall, Dennis P.
通讯作者：
Wall, Dennis P.
DOI：
10.1007/978-3-319-55394-8_1
发表时间：
2017-01-01
期刊：
PRACTICAL GUIDE TO SENTIMENT ANALYSIS
影响因子：
--
作者：
Cambria, Erik;Das, Dipankar;Feraco, Antonio
通讯作者：
Feraco, Antonio