Collaborative Research: RI: Small: Wisdom of Crowds with Machines in the Loop

人类和机器智能的重要性以及它们的互补性已经引起了对人机混合计算系统的渴望，这些系统比任何一个单独都能实现更多。在计算过程中寻求人类输入的人在环计算是一种自然的方法。然而，大多数人在环计算系统都专注于简单的人类输入如何帮助机器更好地执行任务。本研究采取相反的视角，关注以人为中心的领域-群体智慧-并研究如何让机器参与循环可以提高利用群体智慧的效率。一个关键的挑战是直接评估群众贡献的质量。这项研究解决了从人群中获得高质量贡献的问题，尽管缺乏此类评估的数据。该项目旨在更准确和更强大地使用群众贡献，广泛应用于商业（例如群众转录和翻译，以及在线评论），科学（例如公民科学，机器学习，以及会议和期刊的同行评审），教育（例如同行评分）和其他领域。 本研究调查了两个核心问题，以挖掘人群的智慧在具有挑战性的，但现实的，无验证和无监督的设置，没有地面真相，解决两个关键的研究问题：（1）如何从（潜在的战略）人群成员引出高质量的信息;（2）如何汇总引出的信息，形成一个高质量的，集体的意见。缺乏通过基本事实进行的验证对机制设计者提出了一个挑战，即如何调整激励机制以促进启发。这也意味着设计者不知道在给定异质贡献的聚合中谁的信息应该权重更高。本研究发展了一个理论基础的框架，启发和聚合的设置没有验证。它将机器学习方法用于设计启发和聚合机制，以实现众包应用的可证明保证，重点是启发信息的质量和聚合意见的质量。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The importance of both human and machine intelligence and their complementarity has given rise to the aspiration for human-machine hybrid computing systems that achieve more than either could alone. Human-in-the-loop computing, where human inputs are sought during the computation process, is a natural approach. However, most human-in-the-loop computing systems focus on how simple human inputs can help machines to better perform their tasks. This research takes the opposite perspective by focusing on a human-centered domain---the wisdom of crowds---and studies how having machines in the loop can improve the efficacy of harnessing the wisdom of crowds. A key challenge is directly evaluating the quality of crowd contributions. This research tackles the problem of obtaining high-quality contributions from the crowd despite the lack of data for such evaluations. This project seeks to make more accurate and robust use of crowd contributions in a broad spectrum of applications in business (e.g. crowd transcription and translation, and online reviews), sciences (e.g. citizen sciences, machine learning, and peer reviews for conferences and journals), education (e.g. peer grading) and other areas. This research investigates two core problems for tapping into the wisdom of crowds in the challenging, yet realistic, non-verification and unsupervised setting where no ground truth is available, addressing two key research questions: (1) how to elicit high-quality information from (potentially strategic) crowd members; and (2) how to aggregate the elicited information to form a high-quality, collective opinion. Lack of verification via ground truth presents a challenge for the mechanism designer to align incentives for elicitation. It also means that the designer does not know whose information should be weighted higher in aggregation given heterogeneous contributions. This research develops a theoretically grounded framework for elicitation and aggregation for settings without verification. It incorporates machine learning methods for the design of elicitation and aggregation mechanisms to achieve provable guarantees for the crowdsourcing applications, with a focus on the quality of elicited information and the quality of the aggregated opinion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.