Collaborative Research: RI: Small: Wisdom of Crowds with Machines in the Loop

人类和机器智能的重要性及其互补性引发了对人机混合计算系统的渴望，这种系统取得的成就比任何一个单独实现的都要多。人在环计算是一种自然的方法，在计算过程中寻求人的输入。然而，大多数人在环路计算系统关注的是简单的人类输入如何帮助机器更好地执行任务。这项研究采取了相反的观点，聚焦于以人为中心的领域-群体智慧-并研究了在循环中使用机器如何提高利用群体智慧的效率。一个关键的挑战是直接评估大众贡献的质量。这项研究解决的问题是，尽管缺乏此类评估的数据，但仍能从人群中获得高质量的贡献。该项目寻求在商业(例如，群体转录和翻译以及在线评论)、科学(例如，公民科学、机器学习以及对会议和期刊的同行评论)、教育(例如，同行评分)和其他领域的广泛应用中更准确和更有力地使用群体贡献。这项研究调查了在没有基本事实的具有挑战性的、但现实的、非验证的和无监督的环境中挖掘群体智慧的两个核心问题，解决了两个关键的研究问题：(1)如何从(潜在的战略)群体成员那里获得高质量的信息；(2)如何收集获得的信息以形成高质量的集体意见。缺乏通过基本事实进行的核实给机制设计者带来了一个挑战，即如何调整启发式激励措施。这还意味着，在给定异类贡献的情况下，设计者不知道谁的信息在聚合中的权重应该更高。本研究为未经验证的情景设计了一个理论上扎根的启发式和聚合框架。它结合了机器学习方法来设计启发和聚合机制，以实现众包应用程序的可证明保证，重点放在获得的信息的质量和聚合意见的质量上。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The importance of both human and machine intelligence and their complementarity has given rise to the aspiration for human-machine hybrid computing systems that achieve more than either could alone. Human-in-the-loop computing, where human inputs are sought during the computation process, is a natural approach. However, most human-in-the-loop computing systems focus on how simple human inputs can help machines to better perform their tasks. This research takes the opposite perspective by focusing on a human-centered domain---the wisdom of crowds---and studies how having machines in the loop can improve the efficacy of harnessing the wisdom of crowds. A key challenge is directly evaluating the quality of crowd contributions. This research tackles the problem of obtaining high-quality contributions from the crowd despite the lack of data for such evaluations. This project seeks to make more accurate and robust use of crowd contributions in a broad spectrum of applications in business (e.g. crowd transcription and translation, and online reviews), sciences (e.g. citizen sciences, machine learning, and peer reviews for conferences and journals), education (e.g. peer grading) and other areas. This research investigates two core problems for tapping into the wisdom of crowds in the challenging, yet realistic, non-verification and unsupervised setting where no ground truth is available, addressing two key research questions: (1) how to elicit high-quality information from (potentially strategic) crowd members; and (2) how to aggregate the elicited information to form a high-quality, collective opinion. Lack of verification via ground truth presents a challenge for the mechanism designer to align incentives for elicitation. It also means that the designer does not know whose information should be weighted higher in aggregation given heterogeneous contributions. This research develops a theoretically grounded framework for elicitation and aggregation for settings without verification. It incorporates machine learning methods for the design of elicitation and aggregation mechanisms to achieve provable guarantees for the crowdsourcing applications, with a focus on the quality of elicited information and the quality of the aggregated opinion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.