RI: Small: Actively Learning From The Crowd

如何从少数且部分正确的答案中回答大量问题？这个问题是标记大量未标记数据的核心，这在机器学习和数据科学中很常见。这也是通过调查了解人们的偏好和物品质量的核心。解决这个问题的一个流行方法是通过众包平台向大量的人支付少量的钱，让他们在互联网上回答问题，从而将标签或学习任务众包。然而，由于人们的能力和问题的难度不同，工作者的回答质量差异很大。考虑到回答的不确定性，每个问题被分配给多人，他们的回答被汇总。然而，分配过程往往是不可知论的人的能力和问题的困难，因为这两者都是未知的先验。该项目将开发适应人和问题的算法，从而大大减少机器学习算法良好运行所需的响应数量，并使调查具有信息性。除了研究目标之外，研究人员还将通过将该项目的部分内容整合到研究生课程中，促进本科生研究，并通过举办跨学科机器学习研讨会来促进跨学科交流，以实现教育目标。众包的核心优化问题是通过只分配少量任务给人们（或工人），以最小的成本获得对最终答案的信心。直觉上，这可以通过向最有资格回答这个问题的工人提出一个问题来完成。该项目将为众标和众包制定有效和实用的主动方案，自适应地选择向哪个工人提出哪个问题。对于每个算法，该项目将证明相应的严格的计算和统计问题实例相关的性能保证，而不是最坏情况下的性能保证。理论结果将辅以实际的开源实现和真实世界数据的实验。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

How can a large number of questions be answered from few and only partially correct responses? This problem lies at the heart of labeling large collections of unlabeled data, which are common in machine learning and data science. It also lies at the heart of learning about preferences of people and quality of items by carrying out surveys. A popular approach to address this problem is to crowdsource the labeling or learning task by paying a large number of people small amounts of money to answer questions on the internet through a crowdsourcing platform. However, the quality of the workers responses varies significantly due to different abilities of the people and difficulties of the questions. To account for the uncertainty of the responses, each question is assigned to multiple people and their responses are aggregated. However, the assignment process is often agnostic to the peoples abilities and questions difficulties, since both are unknown a priori. This project will develop algorithms that adapt to the people and questions and thereby significantly reduces the number of responses required to enable machine learning algorithms to perform well and surveys to be informative. Besides the research objectives, the researchers will pursue educational objectives by integrating parts of this project into a graduate class, promoting undergraduate research, and fostering exchange across disciplines by running an interdisciplinary machine learning seminar. The core optimization problem in crowdsourcing is to achieve confidence in the final answers at minimal cost, by assigning only few tasks to the people (or workers). Intuitively, that can be accomplished by only posing a question to the workers best qualified to answer that question. This project will develop efficient and practical active schemes for crowdlabeling and crowdsourcing that adaptively choose which question to pose to which worker. For each algorithm, the project will prove corresponding rigorous computational and statistical problem-instance dependent performance guarantees, as opposed to worst-case performance guarantees. The theoretical results will be complemented with practical open-source implementations and experiments on real-world data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.