Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets
Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets
复制标题
DOI：
10.1145/3593013.3594114
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Teanna Barrett;Quan Ze Chen;Amy X. Zhang
中科院分区：
文献类型：
--
作者：
Teanna Barrett;Quan Ze Chen;Amy X. Zhang
为了研究在分析人类图像的计算机视觉系统中观察到的种族差异，研究人员转向肤色作为比种族元数据更客观的注释，以进行公平性评估。然而，肤色注释过程的当前状态是高度变化的。例如，研究人员使用一系列未经测试的量表和肤色类别，注释程序不明确，对不确定性的分析也不充分。此外，很少有人注意到参与注释过程的人的位置性，包括设计师和注释者，以及美国肤色的历史和社会学背景。我们的工作是第一次调查肤色标注过程作为一个社会技术项目。我们调查了最近的肤色标注程序，并进行了标注实验，以研究肤色的主观理解是如何嵌入到肤色标注程序。我们的系统性文献综述揭示了肤色和种族之间的未经询问的关联，以及在计算机视觉评估中肤色注释的当前程序中分析注释者不确定性的有限努力。我们的实验表明，注释过程中的设计决策，例如肤色尺度呈现的顺序或图像中的附加上下文（即，面部的存在）显著地影响了所得到的注释者间一致性和肤色注释的个体不确定性。我们呼吁在使用肤色进行评估的程序的设计、分析和记录方面具有更大的自反性。
To investigate the well-observed racial disparities in computer vision systems that analyze images of humans, researchers have turned to skin tone as a more objective annotation than race metadata for fairness performance evaluations. However, the current state of skin tone annotation procedures is highly varied. For instance, researchers use a range of untested scales and skin tone categories, have unclear annotation procedures, and provide inadequate analyses of uncertainty. In addition, little attention is paid to the positionality of the humans involved in the annotation process—both designers and annotators alike—and the historical and sociological context of skin tone in the United States. Our work is the first to investigate the skin tone annotation process as a sociotechnical project. We surveyed recent skin tone annotation procedures and conducted annotation experiments to examine how subjective understandings of skin tone are embedded in skin tone annotation procedures. Our systematic literature review revealed the uninterrogated association between skin tone and race and the limited effort to analyze annotator uncertainty in current procedures for skin tone annotation in computer vision evaluation. Our experiments demonstrated that design decisions in the annotation procedure such as the order in which the skin tone scale is presented or additional context in the image (i.e., presence of a face) significantly affected the resulting inter-annotator agreement and individual uncertainty of skin tone annotations. We call for greater reflexivity in the design, analysis, and documentation of procedures for evaluation using skin tone.