CAREER: Fundamentals of Learning from People with Applications to Peer Review

在各种各样的应用程序中——比如同行评审、推荐系统、招聘、大学录取、同行评分、a /B测试和众包——从人们那里获取和处理数据是很常见的。这样的数据通常会受到校准错误、主观性、战略行为和偏见等问题的影响。这些问题在应用程序（如上面列出的那些应用程序）中被进一步放大，其中数据包括人们对一组项目的评估，其中每个人只评估项目的一个子集，每个项目只由一个子集的人评估。这些问题降低了这些应用程序的整体质量，也导致了对一些参与者的不公平。例如，来自人们的数据通常与某些人口统计数据有关；评价者的主观意见或严格/宽松会导致不公平；一些参与者可能沉迷于对整个系统有害的战略行为。该项目将设计算法，从人们那里获取数据，并以最大限度地减轻这些问题的方式进行处理。该项目将特别关注学术研究的同行评审应用。它将通过对依赖于人们数据的应用程序的研究成果，推动积极的政策变化，以及协同的教育活动，对现实世界产生重大影响。该项目将从三个方面解决向他人学习时的校准错误、主观性、战略行为和偏见问题。首先，利用信息论和统计学的工具，它将确定这些问题可以缓解的程度的基本限制。其次，它将开发算法，可以证明达到（或接近）这些限制，并且计算效率也很高。这方面的研究将使用机器学习和统计学，博弈论和社会选择理论的工具。最后，该项目将把理论转化为实践者的有用工具包，并推动积极的政策变革。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In a wide variety of applications -- such as peer review, recommender systems, hiring, college admissions, peer grading, A/B testing, and crowdsourcing -- it is common to elicit and process data from people. Such data often suffer from issues such as miscalibration, subjectivity, strategic behavior, and biases. These issues are further amplified in applications (such as those listed above) in which the data comprises evaluations of a set of items by people, where every person evaluates only a subset of items and every item is evaluated by only a subset of people. These issues degrade the overall quality of these applications and also lead to unfairness towards some of its participants. For example, data from people often have biases pertaining to certain demographics; subjective opinions or strictness/leniency of the human evaluators can lead to unfairness; some participants may indulge in strategic behavior which can be detrimental to the overall system. This project will design algorithms for eliciting data from people and processing it in a manner that mitigates these issues to the maximum possible extent. The project will have a particular focus on the application of peer review of scholarly research. It will make a significant real-world impact through the research outcomes for applications that depend on data from people, outreach to drive positive policy changes, and synergistic educational activities.This project will address the issues of miscalibration, subjectivity, strategic behavior and biases in learning from people along three fronts. First, using tools from information theory and statistics, it will establish the fundamental limits on the extent to which these problems can be mitigated. Second, it will develop algorithms that will provably achieve (or approach) these limits, and are also computationally efficient. The research on this front will employ tools from machine learning and statistics, game theory and social choice theory. Finally, the project will transform the theory into a useful toolkit for practitioners, as well as outreach towards driving positive policy changes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.