权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

RI: Small: Improving Crowd-Sourced Annotation by Autonomous Intelligent Agents

RI：小型：通过自主智能代理改进众包注释

基本信息

批准号：
1420667
负责人：
Daniel Weld
金额：
$ 46万
依托单位：
University of Washington
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2014
资助国家：
美国
起止时间：
2014-08-01 至 2018-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1420667&HistoricalAwards=false
关键词：
RI Small Improving Crowd Sourced

项目摘要

Supervised machine learning methods are arguably the greatest success story for Artificial Intellitence with a deep underlying theory and applications ranging from medical diagnosis and scientific data analysis to ecommerce recommender systems and credit-card fraud detection. Unfortunately, all these methods require labeled training data, which has been annotated by a human --- a time consuming and extremely expensive process. This project will use automated decision theory to control the annotation process, saving significant amounts of human labor and extending the practical use of machine learning to a much broader array of societal problems. Specifically, the methods address the case where labeled data is crowd-sourced by a large number of human annotators whose skill and error rates are variable. The project develops new control algorithms that let the learner efficiently ask specific workers to label (or redundantly re-label) specific examples. To test the practicality of their methods, the PIs build and conduct studies with the Information Omnivore, a fully autonomous agent that optimizes the annotation of natural language processing (NLP) training data. By continuously posing questions to paid workers and volunteer citizen-scientists, the Omnivore 1) will learn which problems are hard and which are easy, 2) will learn about the skills of the various workers, 3) and will decide questions to ask which workers in order to maximize the accuracy of the learned model given scare human help. Besides contributing to the science of automated control, the Omnivore will generate labeled training data for two important NLP problems: named entity linking (NEL) and information extraction (IE), greatly helping the community of NLP researchers. Furthermore, the researchers plan a number of outreach efforts, including curriculum development, participation in the K12 Paws on Science program at the Pacific Science Center and interaction with the diverse students comprising the Washington STate Academic RedShirt (STARS) in Engineering program. The specific algorithms proposed by the PIs are notable in several respects. Their decision-theoretic optimization framework operationalizes intuitions like (1) one should assign more or better workers to hard problems and (2) one should redirect effort away from easy questions or from tasks that are too hard to solve. Automating this reasoning is hard because problem difficulty and worker skill are latent variables and thus the agent must confront an exploration / exploitation tradeoff as it balances actions that enable it to learn about the capabilities of workers with the ultimate goal of producing quality annotations. The PIs consider two cases: Task Allocation for Annotation Accuracy tries to maximize the overall annotation accuracy of a fixed size data set through batch assignment of workers to tasks. Re-Active Learning seeks instead to directly construct an accurate ML classifier through a balanced mix of annotator requests to re-label old or label new examples. In both cases they propose a model based on decision-theoretic methods (e.g., partially-observable Markov decision processes (POMDPs) and multi-armed bandits). The PIs propose to integrate their methods in the Information Omnivore, a long-lived software agent that integrates planning and execution, acts in the real world, and learns a model of its environment. The Omnivore will allow large-scale latitudinal studies of their algorithms, and as a byproduct will generate NLP training data that will greatly assist a large community of other researchers.

监督式机器学习方法可以说是人工智能最成功的案例，它有着深厚的基础理论和应用，从医疗诊断和科学数据分析到电子商务推荐系统和信用卡欺诈检测。不幸的是，所有这些方法都需要有标记的训练数据，这些数据都是由人工注释的，这是一个耗时且极其昂贵的过程。该项目将使用自动化决策理论来控制标注过程，节省大量人力，并将机器学习的实际应用扩展到更广泛的社会问题。具体地说，这些方法解决了标记数据由大量人工注释者进行众包的情况，这些注释者的技能和错误率是可变的。该项目开发了新的控制算法，让学习者有效地要求特定的工作人员标记（或冗余重新标记）特定的示例。为了测试其方法的实用性，pi使用信息杂食者（Information Omnivore）构建并进行研究，这是一个完全自主的代理，可以优化自然语言处理（NLP）训练数据的注释。通过不断地向付费工人和志愿公民科学家提出问题，Omnivore 1)将了解哪些问题很难，哪些问题容易，2)将了解各种工人的技能，3)将决定向哪些工人提问，以便在人类帮助下最大限度地提高学习模型的准确性。除了为自动控制科学做出贡献外，Omnivore将为两个重要的NLP问题：命名实体链接（NEL）和信息提取（IE）生成标记训练数据，极大地帮助了NLP研究人员。此外，研究人员还计划开展一些推广工作，包括课程开发、参与太平洋科学中心的K12科学之爪项目，以及与华盛顿州工程项目的不同学生进行互动。pi提出的具体算法在几个方面是值得注意的。他们的决策理论优化框架实现了这样的直觉：(1)应该分配更多或更好的工人来解决难题；(2)应该将精力从容易的问题或难以解决的任务中转移出来。自动化这种推理是很困难的，因为问题难度和工人技能是潜在的变量，因此代理必须面对探索/利用的权衡，因为它要平衡使它能够了解工人的能力和产生高质量注释的最终目标的行动。pi考虑两种情况：注释准确性的任务分配试图通过将工作人员批量分配给任务来最大化固定大小数据集的总体注释准确性。相反，重新主动学习寻求通过平衡混合注释器请求来重新标记旧的或标记新的示例，直接构建一个准确的ML分类器。在这两种情况下，他们提出了一个基于决策理论方法的模型（例如，部分可观察马尔可夫决策过程（pomdp）和多武装强盗）。pi建议将他们的方法集成到信息杂食者中，信息杂食者是一个长期存在的软件代理，它集成了计划和执行，在现实世界中行动，并学习其环境的模型。Omnivore将允许对他们的算法进行大规模的纵向研究，并且作为副产品将产生NLP训练数据，这将极大地帮助其他研究人员的大型社区。