CRII: CIF: Crowdsourcing-aware Learning

CRII：CIF：众包意识学习

• 批准号: 1755656
• 负责人: Nihar Shah
• 金额: $ 17.49万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-04-01 至 2020-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1755656&HistoricalAwards=false
• 关键词: CRII; CIF; Crowdsourcing; aware; Learning

Machine learning has significantly advanced the state of the art in a variety of applications. These successes have required massive labeled datasets for training machine learning algorithms. The collection of these labeled datasets usually involves human annotation. For instance, the training labels for supervised learning algorithms are often obtained through "crowdsourcing" where people label the data over the Internet in exchange for monetary incentives. Most learning algorithms, however, are agnostic of this human-labeling process. This project designs improved learning algorithms by incorporating the "human" aspect of the data collection process in the machine learning objective.In more detail, this project considers supervised binary classification tasks where the labels for the training data are obtained from people. The research involves design of learning algorithms that jointly consider the human collection process -- including the interfaces and incentives available to the human labelers -- and the overall learning objective. Theoretical guarantees of optimality are derived and compared with guarantees for algorithms which are agnostic of the human component. The algorithms and guarantees are based on models of human behavior from psychology, such as permutation-based models, that allow for maximal accuracy while making minimal assumptions on how the human labelers behave. The theoretical results are corroborated with practical implementations (open sourced) and real-world experiments (data freely available online).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习在各种应用中显着推进了最先进的技术。这些成功需要大量的标记数据集来训练机器学习算法。这些标记数据集的收集通常涉及人工注释。例如，监督学习算法的训练标签通常通过“众包”获得，其中人们通过互联网标记数据以换取金钱激励。然而，大多数学习算法对这种人类标记过程是不可知的。该项目通过将数据收集过程中的"人"方面纳入机器学习目标来设计改进的学习算法。更详细地说，该项目考虑了监督二进制分类任务，其中训练数据的标签是从人那里获得的。该研究涉及学习算法的设计，这些算法共同考虑人类收集过程-包括人类标签人员可用的接口和激励措施-以及整体学习目标。最优性的理论保证推导和保证算法是不可知的人的组成部分。算法和保证基于心理学中的人类行为模型，例如基于置换的模型，这些模型允许最大的准确性，同时对人类标签人员的行为做出最小的假设。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

On Testing for Biases in Peer Review

• 发表时间: 2019-12
• 作者: Ivan Stelmakh;Nihar B. Shah;Aarti Singh

On Strategyproof Conference Peer Review

• DOI: 10.24963/ijcai.2019/87
• 发表时间: 2018-06
• 作者: Yichong Xu;H. Zhao;Xiaofei Shi;Nihar B. Shah

Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings

你的 2 是我的 1，你的 3 是我的 9：处理评级中的任意错误校准

• 发表时间: 2019
• 期刊: AAMAS Conference proceedings
• 作者: Wang, J;Shah, N
• 通讯作者: Shah, N

Low Permutation-rank Matrices: Structural Properties and Noisy Completion

低排列秩矩阵：结构属性和噪声完成

• 发表时间: 2019
• 期刊: Journal of machine learning research
• 影响因子: 6
• 作者: Shah, N;Balakrishnan, S;Wainwright, M
• 通讯作者: Wainwright, M

Active ranking from pairwise comparisons and when parametric assumptions do not help

• DOI: 10.1214/18-aos1772
• 发表时间: 2016-06
• 期刊: The Annals of Statistics
• 作者: Reinhard Heckel;Nihar B. Shah;K. Ramchandran;M. Wainwright