Collaborative Research: RI: Small: Wisdom of Crowds with Machines in the Loop

人类和机器智能的重要性以及它们的互补性已经引起了人们对人机混合计算系统的渴望，这种系统比任何一方单独实现的都要多。人在循环计算，即在计算过程中寻找人的输入，是一种自然的方法。然而，大多数人在循环计算系统关注的是简单的人类输入如何帮助机器更好地执行任务。这项研究采取了相反的观点，专注于以人为中心的领域——群体智慧——并研究了让机器参与其中如何提高利用群体智慧的效率。一个关键的挑战是直接评估群众贡献的质量。本研究解决了在缺乏此类评估数据的情况下从人群中获得高质量贡献的问题。该项目旨在在商业（如群体转录和翻译、在线评论）、科学（如公民科学、机器学习、会议和期刊的同行评议）、教育（如同行评分）和其他领域的广泛应用中更准确、更有力地利用群体贡献。本研究探讨了在具有挑战性的、现实的、无验证和无监督的、没有基础真理的环境中挖掘群体智慧的两个核心问题，解决了两个关键的研究问题：(1)如何从（潜在的）群体成员那里获得高质量的信息；(2)如何将收集到的信息聚合起来，形成高质量的集体意见。缺乏通过实际事实的验证对机制设计者提出了挑战，使激励机制与启发机制保持一致。这也意味着设计师不知道在给定异构贡献的聚合中，谁的信息应该权重更高。本研究开发了一个理论基础的框架，用于未经验证的设置的启发和聚合。它结合了机器学习方法来设计引出和聚合机制，以实现众包应用程序的可证明保证，重点是引出信息的质量和聚合意见的质量。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The importance of both human and machine intelligence and their complementarity has given rise to the aspiration for human-machine hybrid computing systems that achieve more than either could alone. Human-in-the-loop computing, where human inputs are sought during the computation process, is a natural approach. However, most human-in-the-loop computing systems focus on how simple human inputs can help machines to better perform their tasks. This research takes the opposite perspective by focusing on a human-centered domain---the wisdom of crowds---and studies how having machines in the loop can improve the efficacy of harnessing the wisdom of crowds. A key challenge is directly evaluating the quality of crowd contributions. This research tackles the problem of obtaining high-quality contributions from the crowd despite the lack of data for such evaluations. This project seeks to make more accurate and robust use of crowd contributions in a broad spectrum of applications in business (e.g. crowd transcription and translation, and online reviews), sciences (e.g. citizen sciences, machine learning, and peer reviews for conferences and journals), education (e.g. peer grading) and other areas. This research investigates two core problems for tapping into the wisdom of crowds in the challenging, yet realistic, non-verification and unsupervised setting where no ground truth is available, addressing two key research questions: (1) how to elicit high-quality information from (potentially strategic) crowd members; and (2) how to aggregate the elicited information to form a high-quality, collective opinion. Lack of verification via ground truth presents a challenge for the mechanism designer to align incentives for elicitation. It also means that the designer does not know whose information should be weighted higher in aggregation given heterogeneous contributions. This research develops a theoretically grounded framework for elicitation and aggregation for settings without verification. It incorporates machine learning methods for the design of elicitation and aggregation mechanisms to achieve provable guarantees for the crowdsourcing applications, with a focus on the quality of elicited information and the quality of the aggregated opinion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.