Efficient and Reproducible Image Annotation for Supervised Deep Learning with Small Data

机器学习是目前最热门、发展最快的技术之一。每个人都想用“深度学习”来开发无所不包的系统：人脸识别、身份跟踪、制造业中的质量控制、视频中的物体跟踪、疾病诊断、精准农业等等。当有非常大量的图像可供机器学习算法学习时，深度学习的效果最好。对于人们来说，用适当的标签来标记汽车、船只或飞机，甚至不同类型的汽车的图像相对容易。创建大型数据集来训练诊断癌症的系统，或计算植物上的花朵数量，挑战要大得多。标记医学图像需要经验丰富的专家。在一株植物上标记所有的花时，很容易遗漏一朵花，或者一朵花被一片叶子或另一朵花部分遮挡，或者在指定花的确切位置时不准确。此外，为了获得足够的带注释的图像以供学习，通常有多个注释器标记一个数据集，但每个图像只由一个注释器进行注解。在这种情况下，我们无法确定不同的注释器是否一致地进行注解，也就是说，我们不能量化注释器之间的协议。注释器的不一致是由于它们的标注存在偏差，并降低了训练数据集的质量。批注任务的难度和给予批注员的指令的性质影响了评分员之间的一致性。如果我们能够找到在多个注释器上获得更一致的标注的方法，那么训练数据集的质量将会提高，因此学习系统的性能也会提高。拟议的研究计划将研究如何从具有更高注释器间和注释器内一致性的注释器获得更高质量的注释器。我们将创建增强的注释工具，提供特定于问题的半自动化来帮助注释员，并量化由此产生的对注释员协议和经过培训的系统性能的好处。我们将量化注释器协议与模型性能之间的关系。我们将探索上下文因素(如注释类型、给出的说明、压力和分心)对注释者一致性的影响程度，并开发减轻其影响的最佳实践。通过研究影响注释器一致性的因素和从带注释的数据集学习的系统的性能，我们将能够开发新的标准化方法，用于在有限数据的情况下训练“深度学习”模型。这将允许更好地预测投资于注释的最佳资源量，减少对反复试验方法的依赖，以获得最佳训练的系统性能，并使成功的机器学习减少对深厚技术专业知识的依赖。

Machine learning is currently one of the hottest and most quickly evolving technologies. Everyone wants to use "deep learning" to develop systems for everything: face recognition, identity tracking, quality control in manufacturing, tracking objects in videos, diagnosing disease, precision agriculture, etc. Deep learning works best when there are very large numbers of images where the "correct answer" is known from which the machine learning algorithm can learn. It is relatively easy for people to tag an image of a car, a boat, or a plane, or even different types of cars with an appropriate label. It is much more challenging to create large datasets to train systems for diagnosing cancer, or counting the number of flowers on a plant. Labeling medical images requires highly experienced experts. When marking all of the flowers on a plant, it is easy to miss one, or for a flower to be partially occluded by a leaf or another flower, or to be imprecise in specifying a flower's exact location. Moreover, in order to obtain enough annotated images to learn from, typically multiple annotators label a dataset but each image is annotated by only one annotator. In such circumstances we have no way of determining whether different annotators are annotating consistently, that is, we can't quantify the inter-annotator agreement. Annotator disagreement arises from biases in their annotations, and degrades the quality of the training dataset. Inter-rater agreement is affected by the difficulty of the annotation task and the nature of the instructions given to annotators. If we can find ways of obtaining more consistent annotations across multiple annotators, training dataset quality will improve, and hence the performance of the learned system also improves. The proposed research program will study how to obtain better quality annotations from annotators with higher inter- and intra-annotator agreement. We will create augmented annotation tools that provide problem-specific semi-automation to assist annotators and quantify the resulting benefits to annotator agreement and trained system performance. We will quantify the relationship between annotator agreement and model performance. We will explore the degree to which contextual factors such as annotation type, instructions given, pressure, and distractions can influence annotator agreement and develop best practices for mitigating their effects. By studying factors that influence annotator agreement and the performance of the systems that are learned from annotated datasets, we will be able to develop new standardized methodologies for training "deep learning" models with limited data. This will allow better prediction of the optimal amount of resources to invest in annotation, reduce the reliance on trial-and-error methods to obtain the best trained system performance, and make successful machine learning less reliant on deep technical expertise.