CAREER: Active Learning in the Real World

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。现代数字数据的爆炸式增长扩大了使用计算学习框架解决现实世界问题的可能性。然而，用标签标注数据，通常包括手动在重要变量上添加信息标签，以训练机器学习模型，在时间、劳动力和人力方面仍然是一个昂贵的过程。主动学习算法通过从大量需要手动标记的未标记数据中自动识别显著样本和范例样本来缓解这一挑战。在传统的主动学习设置中，被称为预言机的标签实体被认为是绝对正确的；也就是说，它们总是为查询的样本提供正确的答案（就类标签而言）。然而，在现实世界的应用程序中，预言机通常是不完美的；它们可能提供不正确的注释，甚至可能不愿意提供任何注释。该项目的总体目标是在具有挑战性的现实世界约束下开发新的主动学习算法，以弥合理想世界和现实世界主动学习之间的差距。这个项目有可能大大减少现实世界人工智能系统（如医疗诊断、安全和监控）设计中的人类注释工作，从而产生重大的社会影响。作为该项目的一部分，研究者计划教育广泛的学生，这将有助于确保一个强大的计算机科学家管道，以满足国家的技术需求；他还将寻求增加女性和未被充分代表的少数族裔学生在STEM相关研究生项目中的数量。这个项目的目标将通过两个研究目标来实现。首先，将开发一种新的主动学习框架，该框架可以联合识别信息丰富的未标记样本以及每个样本的最优标记预测。这将使获得正确标签的概率最大化。其次，将探索新的查询和注释机制，这些机制更容易出错，更容易回答，从而增加获得可靠注释的机会。第二个目标将通过两个创新的查询框架来证明：(i)基于视觉的面部年龄估计的深度主动学习算法，注释者只需要提供给定范围内一个人年龄的最佳估计上限和下限，而不是可能难以估计的确切年龄；（ii）一个用于多类分类的深度主动学习框架，在这个框架中，除了最上面的选择之外，人类注释者可以根据给定的查询提供替代标签。这在注释器不知道所有类的应用程序中很有用，并且对于给定的示例可能有多个类选择。还将进行广泛的用户研究，以了解所建议的解决方案的优点和缺点。这项研究将为开发新的主动学习框架打开大门，这些框架旨在应对具有挑战性的现实世界的限制。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The explosive growth of digital data in the modern era has expanded the possibilities of solving real-world problems using computational learning frameworks. However, annotating the data with labels, which often includes manually adding informative tags on variables of importance in order to train a machine learning model has remained an expensive process in terms of time, labor and human expertise. Active learning algorithms alleviate this challenge by automatically identifying the salient and exemplar samples from large amounts of unlabeled data, which need to be labeled manually. In a traditional active learning setup, the labeling entities, called oracles, are assumed to be infallible; that is, they always provide the correct answers (in terms of class labels) to the queried samples. However, in real-world applications, the oracles are often imperfect; they may provide incorrect annotations and may even be reluctant to provide any annotations. The overarching goal of this project is to develop novel active leaning algorithms under such challenging, real-world constraints, in an effort to bridge the gap between ideal world and real-world active learning. This project has the potential to tremendously reduce human annotation effort in the design of real-world AI systems (such as medical diagnosis, security and surveillance), thereby creating a significant societal impact. As part of this project, the investigator plans to educate a wide spectrum of students which will help to ensure a strong pipeline of computer scientists to satisfy the technical needs of the nation; he will also seek to increase the number of females and students from underrepresented minorities in graduate STEM related programs.The goals of this project will be realized through two research objectives. First, a novel active learning framework will be developed, which can jointly identify the informative unlabeled samples together with the optimal labeling oracles for each sample. This should maximize the probability of obtaining the correct label. Second, novel query and annotation mechanisms will be explored which are less error-prone and easier to answer, thereby increasing the chances of obtaining reliable annotations. The second objective will be demonstrated through two innovative query frameworks: (i) a deep active learning algorithm for vision-based facial age estimation, where the annotators merely need to provide the best estimated upper and lower bounds on the age of a person within a given span, rather than the exact age which may be difficult to estimate; and, (ii) a deep active learning framework for multiclass classification, where the human annotators are allowed to provide alternative labels in response to a given query apart from the topmost choice. This is useful in applications where the annotators are not cognizant about all the classes and may have more than one class choices in mind for a given sample. Extensive user studies will also be conducted to understand the benefits and drawbacks of the proposed solutions. This research will open the door to the development of novel active learning frameworks which are designed to operate in the presence of challenging, real-world constraints.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.