Towards Professional Level Crowd Annotation of Expert Domain Data
Towards Professional Level Crowd Annotation of Expert Domain Data
复制标题
DOI：
10.1109/cvpr52729.2023.00309
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Pei Wang;N. Vasconcelos
中科院分区：
文献类型：
--
作者：
Pei Wang;N. Vasconcelos
专家领域的图像识别通常是细粒度的，需要专家标记，这是昂贵的。这限制了数据集的大小和学习系统的准确性。为了应对这一挑战，我们考虑通过众包来注释专家数据。这被表示为专业水平人群 (POSER) 注释。提出了一种基于半监督学习（SSL）并表示为人工过滤的 SSL（SSL-HF）的新方法。它是一种人机循环 SSL 方法，其中众包工作人员充当伪标签过滤器，取代了最先进的 SSL 方法所使用的不可靠的置信阈值。为了使非专家能够进行注释，通过正面和负面的示例集隐式指定类别，并通过深思熟虑的解释进行增强，从而突出显示类别模糊性的区域。通过这种方式，SSL-HF 利用人类强大的低样本学习和置信估计能力来创建直观但有效的标记体验。实验表明，SSL-HF 在多个基准测试中显着优于各种替代方法。
Image recognition on expert domains is usually fine-grained and requires expert labeling, which is costly. This limits dataset sizes and the accuracy of learning systems. To address this challenge, we consider annotating expert data with crowdsourcing. This is denoted as PrOfeSsional lEvel cRowd (POSER) annotation. A new approach, based on semi-supervised learning (SSL) and denoted as SSL with human filtering (SSL-HF) is proposed. It is a human-in-the-loop SSL method, where crowd-source workers act as filters of pseudo-labels, replacing the unreliable confidence thresholding used by state-of-the-art SSL methods. To enable annotation by non-experts, classes are specified implicitly, via positive and negative sets of examples and augmented with deliberative explanations, which highlight regions of class ambiguity. In this way, SSL-HF leverages the strong low-shot learning and confidence estimation ability of humans to create an intuitive but effective labeling experience. Experiments show that SSL-HF significantly outperforms various alternative approaches in several benchmarks.