CRII: CIF: Crowdsourcing-aware Learning

机器学习在各种应用中显著地推动了技术的发展。这些成功需要大量标记数据集来训练机器学习算法。这些标记数据集的收集通常涉及人工注释。例如，监督学习算法的训练标签通常是通过“众包”获得的，人们在互联网上给数据贴上标签，以换取金钱奖励。然而，大多数学习算法都不知道这种人类标记过程。该项目通过将数据收集过程中的“人类”方面纳入机器学习目标，设计了改进的学习算法。更详细地说，这个项目考虑了监督二分类任务，其中训练数据的标签是从人那里获得的。该研究涉及学习算法的设计，这些算法共同考虑了人类收集过程——包括人类标注者可用的界面和激励——以及总体学习目标。推导了最优性的理论保证，并与不可知人为因素的算法的保证进行了比较。算法和保证基于心理学中的人类行为模型，例如基于排列的模型，这些模型允许最大的准确性，同时对人类标注者的行为进行最小的假设。理论结果与实际实现（开源）和真实世界的实验（在线免费提供数据）相证实。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning has significantly advanced the state of the art in a variety of applications. These successes have required massive labeled datasets for training machine learning algorithms. The collection of these labeled datasets usually involves human annotation. For instance, the training labels for supervised learning algorithms are often obtained through "crowdsourcing" where people label the data over the Internet in exchange for monetary incentives. Most learning algorithms, however, are agnostic of this human-labeling process. This project designs improved learning algorithms by incorporating the "human" aspect of the data collection process in the machine learning objective.In more detail, this project considers supervised binary classification tasks where the labels for the training data are obtained from people. The research involves design of learning algorithms that jointly consider the human collection process -- including the interfaces and incentives available to the human labelers -- and the overall learning objective. Theoretical guarantees of optimality are derived and compared with guarantees for algorithms which are agnostic of the human component. The algorithms and guarantees are based on models of human behavior from psychology, such as permutation-based models, that allow for maximal accuracy while making minimal assumptions on how the human labelers behave. The theoretical results are corroborated with practical implementations (open sourced) and real-world experiments (data freely available online).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.