Understanding and Evaluating Racial Biases in Image Captioning
Understanding and Evaluating Racial Biases in Image Captioning
复制标题
DOI：
10.1109/iccv48922.2021.01456
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Dora Zhao;Angelina Wang;Olga Russakovsky
中科院分区：
文献类型：
--
作者：
Dora Zhao;Angelina Wang;Olga Russakovsky
图像字幕是基准视觉推理并为有视力障碍的人提供可访问性的重要任务。但是，就像在许多机器学习环境中一样，社会偏见可以以不良的方式影响图像字幕。在这项工作中，我们研究图像字幕内的偏置传播途径，专门针对可可数据集。先前的工作已经使用自动衍生的性别标签分析了字幕中的性别偏见；在这里，我们使用手动注释检查种族和交叉偏见。我们的第一个贡献是在获得IRB批准后注释28,315人的28,315人的性别和肤色。使用这些注释，我们比较手动和自动生成图像标题中存在的种族偏见。我们在标题表现，情感和较浅皮肤的人的图像之间进行了选择差异。此外，我们发现与较旧的字幕相比，现代字幕系统中这些差异的幅度更大，因此导致人们担心如果没有适当的考虑和缓解这些差异，这些差异只会变得越来越普遍。代码和数据可从https://princetonvisualai.github.io/imagecaptioning-bias/获得。
Image captioning is an important task for benchmarking visual reasoning and for enabling accessibility for people with vision impairments. However, as in many machine learning settings, social biases can influence image captioning in undesirable ways. In this work, we study bias propagation pathways within image captioning, focusing specifically on the COCO dataset. Prior work has analyzed gender bias in captions using automatically-derived gender labels; here we examine racial and intersectional biases using manual annotations. Our first contribution is in annotating the perceived gender and skin color of 28,315 of the depicted people after obtaining IRB approval. Using these annotations, we compare racial biases present in both manual and automatically-generated image captions. We demonstrate differences in caption performance, sentiment, and word choice between images of lighter versus darker-skinned people. Further, we find the magnitude of these differences to be greater in modern captioning systems compared to older ones, thus leading to concerns that without proper consideration and mitigation these differences will only become increasingly prevalent. Code and data is available at https://princetonvisualai.github.io/imagecaptioning-bias/.