Flexible and scalable annotation tool to develop scene understanding datasets
Flexible and scalable annotation tool to develop scene understanding datasets
复制标题
DOI：
10.1145/3546930.3547499
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Md. Fazle Elahi Khan;Renran Tian;Xiao Luo
中科院分区：
文献类型：
--
作者：
Md. Fazle Elahi Khan;Renran Tian;Xiao Luo
数据驱动的视觉和基于语言的任务的最新进展要求开发训练数据集，丰富了代表人类智能的多种模式。文本和图像数据之间的联系是开发人工智能模型的关键模式之一。视频领域数据集的开发过程需要研究人员和注释人员（专家和非专家）付出很大的努力。研究人员重新设计注释工具，从注释者那里提取知识，以回答新的研究问题。每个新问题都要重复整个过程，这很耗时。然而，自过去十年以来，研究人员和注释者在注释过程中的互动方式几乎没有变化。我们回顾了注释工作流程，并提出了一个可适应和可扩展的注释工具的概念。该理念强调用户的交互性，使标注过程设计无缝、高效。研究人员可以方便地添加新的模式或增加现有的数据集使用该工具。注释器可以有效地将自由格式的文本链接到图像对象。对于进行任何规模的人体实验，该工具支持数据收集，以获得群体基础真理。我们在74名非专业人士的参与下，使用原型工具在两组之间进行了案例研究。我们发现，自由形式的文本与图像对象的交互链接感觉直观，并唤起了一个思维过程，从而产生了高质量的注释。新设计的数据标注质量提高了约35%。在用户体验评估方面，我们收到了25个人关于便利性、UI辅助、可用性和满意度的高于平均水平的积极反馈。
Recent progress in data-driven vision and language-based tasks demands developing training datasets enriched with multiple modalities representing human intelligence. The link between text and image data is one of the crucial modalities for developing AI models. The development process of such datasets in the video domain requires much effort from researchers and annotators (experts and non-experts). Researchers re-design annotation tools to extract knowledge from annotators to answer new research questions. The whole process repeats for each new question which is time-consuming. However, since the last decade, there has been little change in how the researchers and annotators interact with the annotation process. We revisit the annotation workflow and propose a concept of an adaptable and scalable annotation tool. The concept emphasizes its users' interactivity to make annotation process design seamless and efficient. Researchers can conveniently add newer modalities to or augment the extant datasets using the tool. The annotators can efficiently link free-form text to image objects. For conducting human-subject experiments on any scale, the tool supports the data collection for attaining group ground truth. We have conducted a case study using a prototype tool between two groups with the participation of 74 non-expert people. We find that the interactive linking of free-form text to image objects feels intuitive and evokes a thought process resulting in a high-quality annotation. The new design shows ≈ 35% improvement in the data annotation quality. On UX evaluation, we receive above-average positive feedback from 25 people regarding convenience, UI assistance, usability, and satisfaction.