CAREER: Active Learning in the Real World

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。现代数字数据的爆炸性增长扩大了使用计算学习框架解决现实世界问题的可能性。然而，用标签对数据进行注释，通常包括手动在重要变量上添加信息性标签，以训练机器学习模型，这在时间、劳动力和人力专业知识方面仍然是一个昂贵的过程。主动学习算法通过从大量需要人工标记的未标记数据中自动识别显著样本和样本样本来缓解这一挑战。在传统的主动学习设置中，被称为Oracle的标记实体被认为是不会出错的；也就是说，它们总是为查询的样本提供正确的答案(就类标签而言)。然而，在实际应用程序中，Oracle经常是不完美的；它们可能提供不正确的注释，甚至可能不愿提供任何注释。这个项目的总体目标是在这种具有挑战性的现实世界的约束下开发新的主动学习算法，努力弥合理想世界和现实世界主动学习之间的差距。该项目有可能极大地减少在设计真实世界的人工智能系统(如医疗诊断、安全和监视)中的人类注释工作，从而产生重大的社会影响。作为这一项目的一部分，研究人员计划教育广泛的学生，这将有助于确保强大的计算机科学家渠道，以满足国家的技术需求；他还将寻求增加STEM研究生课程中女性和代表不足的少数民族学生的数量。该项目的目标将通过两个研究目标实现。首先，开发了一种新的主动学习框架，该框架可以联合识别信息丰富的未标记样本和每个样本的最优标记预言。这应该最大限度地提高获得正确标签的可能性。其次，将探索新的查询和注释机制，这些机制不太容易出错，更容易回答，从而增加获得可靠注释的机会。第二个目标将通过两个创新的查询框架来演示：(I)用于基于视觉的面部年龄估计的深度主动学习算法，其中注释者只需要提供关于给定跨度内的人的年龄的最佳估计上界和下界，而不是难以估计的确切年龄；以及(Ii)用于多类分类的深度主动学习框架，其中允许人类注释者响应于给定查询而提供除了最上面的选择之外的替代标签。这在注释器不了解所有类的应用程序中很有用，并且对于给定的样本可能有多个类选择。还将进行广泛的用户研究，以了解拟议解决方案的好处和缺点。这项研究将为开发新的主动学习框架打开大门，这些框架旨在在具有挑战性的、现实世界的限制下运行。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The explosive growth of digital data in the modern era has expanded the possibilities of solving real-world problems using computational learning frameworks. However, annotating the data with labels, which often includes manually adding informative tags on variables of importance in order to train a machine learning model has remained an expensive process in terms of time, labor and human expertise. Active learning algorithms alleviate this challenge by automatically identifying the salient and exemplar samples from large amounts of unlabeled data, which need to be labeled manually. In a traditional active learning setup, the labeling entities, called oracles, are assumed to be infallible; that is, they always provide the correct answers (in terms of class labels) to the queried samples. However, in real-world applications, the oracles are often imperfect; they may provide incorrect annotations and may even be reluctant to provide any annotations. The overarching goal of this project is to develop novel active leaning algorithms under such challenging, real-world constraints, in an effort to bridge the gap between ideal world and real-world active learning. This project has the potential to tremendously reduce human annotation effort in the design of real-world AI systems (such as medical diagnosis, security and surveillance), thereby creating a significant societal impact. As part of this project, the investigator plans to educate a wide spectrum of students which will help to ensure a strong pipeline of computer scientists to satisfy the technical needs of the nation; he will also seek to increase the number of females and students from underrepresented minorities in graduate STEM related programs.The goals of this project will be realized through two research objectives. First, a novel active learning framework will be developed, which can jointly identify the informative unlabeled samples together with the optimal labeling oracles for each sample. This should maximize the probability of obtaining the correct label. Second, novel query and annotation mechanisms will be explored which are less error-prone and easier to answer, thereby increasing the chances of obtaining reliable annotations. The second objective will be demonstrated through two innovative query frameworks: (i) a deep active learning algorithm for vision-based facial age estimation, where the annotators merely need to provide the best estimated upper and lower bounds on the age of a person within a given span, rather than the exact age which may be difficult to estimate; and, (ii) a deep active learning framework for multiclass classification, where the human annotators are allowed to provide alternative labels in response to a given query apart from the topmost choice. This is useful in applications where the annotators are not cognizant about all the classes and may have more than one class choices in mind for a given sample. Extensive user studies will also be conducted to understand the benefits and drawbacks of the proposed solutions. This research will open the door to the development of novel active learning frameworks which are designed to operate in the presence of challenging, real-world constraints.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.