CAREER: Fundamentals of Learning from People with Applications to Peer Review

职业：向人学习的基础知识及其在同行评审中的应用

• 批准号: 1942124
• 负责人: Nihar Shah
• 金额: $ 64.9万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-02-01 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1942124&HistoricalAwards=false
• 关键词: CAREER; Fundamentals; Learning; People; Applications

In a wide variety of applications -- such as peer review, recommender systems, hiring, college admissions, peer grading, A/B testing, and crowdsourcing -- it is common to elicit and process data from people. Such data often suffer from issues such as miscalibration, subjectivity, strategic behavior, and biases. These issues are further amplified in applications (such as those listed above) in which the data comprises evaluations of a set of items by people, where every person evaluates only a subset of items and every item is evaluated by only a subset of people. These issues degrade the overall quality of these applications and also lead to unfairness towards some of its participants. For example, data from people often have biases pertaining to certain demographics; subjective opinions or strictness/leniency of the human evaluators can lead to unfairness; some participants may indulge in strategic behavior which can be detrimental to the overall system. This project will design algorithms for eliciting data from people and processing it in a manner that mitigates these issues to the maximum possible extent. The project will have a particular focus on the application of peer review of scholarly research. It will make a significant real-world impact through the research outcomes for applications that depend on data from people, outreach to drive positive policy changes, and synergistic educational activities.This project will address the issues of miscalibration, subjectivity, strategic behavior and biases in learning from people along three fronts. First, using tools from information theory and statistics, it will establish the fundamental limits on the extent to which these problems can be mitigated. Second, it will develop algorithms that will provably achieve (or approach) these limits, and are also computationally efficient. The research on this front will employ tools from machine learning and statistics, game theory and social choice theory. Finally, the project will transform the theory into a useful toolkit for practitioners, as well as outreach towards driving positive policy changes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在各种各样的应用中--比如同行评议、推荐系统、招聘、大学招生、同行评分、A/B测试和众包--从人们那里获取和处理数据是很常见的。这些数据往往会受到诸如错误校准、主观性、战略行为和偏见等问题的影响。这些问题在数据包括人们对一组项目的评估的应用（例如上面列出的那些应用）中被进一步放大，其中每个人仅评估项目的子集，并且每个项目仅由人的子集评估。这些问题降低了这些应用程序的整体质量，也导致了对一些参与者的不公平。例如，来自人们的数据通常具有与某些人口统计学相关的偏见;人类评估者的主观意见或严格/宽松可能导致不公平;一些参与者可能沉迷于可能对整个系统有害的战略行为。该项目将设计算法，用于从人们那里获取数据，并以最大限度地缓解这些问题的方式处理数据。该项目将特别侧重于学术研究同行评议的应用。该项目将通过研究成果对依赖于人的数据的应用程序产生重大的现实影响，通过推广来推动积极的政策变化，以及协同教育活动。该项目将从沿着三个方面解决人们学习中的错误校准，主观性，战略行为和偏见问题。首先，利用信息论和统计学的工具，它将确定这些问题可以减轻的程度的基本限制。其次，它将开发可证明达到（或接近）这些限制的算法，并且在计算上也是有效的。这方面的研究将采用机器学习和统计学，博弈论和社会选择理论的工具。最后，该项目将把理论转化为实践者的有用工具包，并推动积极的政策变化。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Integrating Rankings into Quantized Scores in Peer Review

• DOI: 10.48550/arxiv.2204.03505
• 发表时间: 2022-04
• 期刊: Trans. Mach. Learn. Res.
• 作者: Yusha Liu;Yichong Xu;Nihar B. Shah;Aarti Singh

Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

• DOI: 10.1609/aaai.v35i6.16611
• 发表时间: 2020-10
• 期刊: ArXiv
• 作者: Ivan Stelmakh;Nihar B. Shah;Aarti Singh

Assisting Human Decisions in Document Matching

• DOI: 10.48550/arxiv.2302.08450
• 发表时间: 2023-02
• 期刊: ArXiv
• 作者: Joon Sik Kim;Valerie Chen;Danish Pruthi;Nihar B. Shah;Ameet Talwalkar

A Heuristic for Statistical Seriation

统计序列化的启发式

• 发表时间: 2021
• 期刊: UAI
• 作者: Dhull, Komal;Wang, Jingyan;Shah, Nihar;Li, Yuanzhi;and Ravi, R
• 通讯作者: and Ravi, R

PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

• 发表时间: 2018-06
• 期刊: J. Mach. Learn. Res.
• 作者: Ivan Stelmakh;Nihar B. Shah;Aarti Singh