RI: Small: Improving Crowd-Sourced Annotation by Autonomous Intelligent Agents
RI: Small: Improving Crowd-Sourced Annotation by Autonomous Intelligent Agents
批准号:
1420667
负责人:
Daniel Weld
金额:
$46.0万
依托单位:
依托单位国家:
美国
项目类别:
Standard Grant
财政年份:
2014
资助国家:
美国
项目状态:
已结题
起止时间:
2014-08-01 至 2018-07-31
中文摘要
监督式机器学习方法可以说是人工智能最成功的案例,它有着深厚的基础理论和应用,从医疗诊断和科学数据分析到电子商务推荐系统和信用卡欺诈检测。不幸的是,所有这些方法都需要有标记的训练数据,这些数据都是由人工注释的,这是一个耗时且极其昂贵的过程。该项目将使用自动化决策理论来控制标注过程,节省大量人力,并将机器学习的实际应用扩展到更广泛的社会问题。具体地说,这些方法解决了标记数据由大量人工注释者进行众包的情况,这些注释者的技能和错误率是可变的。该项目开发了新的控制算法,让学习者有效地要求特定的工作人员标记(或冗余重新标记)特定的示例。为了测试其方法的实用性,pi使用信息杂食者(Information Omnivore)构建并进行研究,这是一个完全自主的代理,可以优化自然语言处理(NLP)训练数据的注释。通过不断地向付费工人和志愿公民科学家提出问题,Omnivore 1)将了解哪些问题很难,哪些问题容易,2)将了解各种工人的技能,3)将决定向哪些工人提问,以便在人类帮助下最大限度地提高学习模型的准确性。除了为自动控制科学做出贡献外,Omnivore将为两个重要的NLP问题:命名实体链接(NEL)和信息提取(IE)生成标记训练数据,极大地帮助了NLP研究人员。此外,研究人员还计划开展一些推广工作,包括课程开发、参与太平洋科学中心的K12科学之爪项目,以及与华盛顿州工程项目的不同学生进行互动。pi提出的具体算法在几个方面是值得注意的。他们的决策理论优化框架实现了这样的直觉:(1)应该分配更多或更好的工人来解决难题;(2)应该将精力从容易的问题或难以解决的任务中转移出来。自动化这种推理是很困难的,因为问题难度和工人技能是潜在的变量,因此代理必须面对探索/利用的权衡,因为它要平衡使它能够了解工人的能力和产生高质量注释的最终目标的行动。pi考虑两种情况:注释准确性的任务分配试图通过将工作人员批量分配给任务来最大化固定大小数据集的总体注释准确性。相反,重新主动学习寻求通过平衡混合注释器请求来重新标记旧的或标记新的示例,直接构建一个准确的ML分类器。在这两种情况下,他们提出了一个基于决策理论方法的模型(例如,部分可观察马尔可夫决策过程(pomdp)和多武装强盗)。pi建议将他们的方法集成到信息杂食者中,信息杂食者是一个长期存在的软件代理,它集成了计划和执行,在现实世界中行动,并学习其环境的模型。Omnivore将允许对他们的算法进行大规模的纵向研究,并且作为副产品将产生NLP训练数据,这将极大地帮助其他研究人员的大型社区。
英文摘要
Supervised machine learning methods are arguably the greatest success story for Artificial Intellitence with a deep underlying theory and applications ranging from medical diagnosis and scientific data analysis to ecommerce recommender systems and credit-card fraud detection. Unfortunately, all these methods require labeled training data, which has been annotated by a human --- a time consuming and extremely expensive process. This project will use automated decision theory to control the annotation process, saving significant amounts of human labor and extending the practical use of machine learning to a much broader array of societal problems. Specifically, the methods address the case where labeled data is crowd-sourced by a large number of human annotators whose skill and error rates are variable. The project develops new control algorithms that let the learner efficiently ask specific workers to label (or redundantly re-label) specific examples. To test the practicality of their methods, the PIs build and conduct studies with the Information Omnivore, a fully autonomous agent that optimizes the annotation of natural language processing (NLP) training data. By continuously posing questions to paid workers and volunteer citizen-scientists, the Omnivore 1) will learn which problems are hard and which are easy, 2) will learn about the skills of the various workers, 3) and will decide questions to ask which workers in order to maximize the accuracy of the learned model given scare human help. Besides contributing to the science of automated control, the Omnivore will generate labeled training data for two important NLP problems: named entity linking (NEL) and information extraction (IE), greatly helping the community of NLP researchers. Furthermore, the researchers plan a number of outreach efforts, including curriculum development, participation in the K12 Paws on Science program at the Pacific Science Center and interaction with the diverse students comprising the Washington STate Academic RedShirt (STARS) in Engineering program. The specific algorithms proposed by the PIs are notable in several respects. Their decision-theoretic optimization framework operationalizes intuitions like (1) one should assign more or better workers to hard problems and (2) one should redirect effort away from easy questions or from tasks that are too hard to solve. Automating this reasoning is hard because problem difficulty and worker skill are latent variables and thus the agent must confront an exploration / exploitation tradeoff as it balances actions that enable it to learn about the capabilities of workers with the ultimate goal of producing quality annotations. The PIs consider two cases: Task Allocation for Annotation Accuracy tries to maximize the overall annotation accuracy of a fixed size data set through batch assignment of workers to tasks. Re-Active Learning seeks instead to directly construct an accurate ML classifier through a balanced mix of annotator requests to re-label old or label new examples. In both cases they propose a model based on decision-theoretic methods (e.g., partially-observable Markov decision processes (POMDPs) and multi-armed bandits). The PIs propose to integrate their methods in the Information Omnivore, a long-lived software agent that integrates planning and execution, acts in the real world, and learns a model of its environment. The Omnivore will allow large-scale latitudinal studies of their algorithms, and as a byproduct will generate NLP training data that will greatly assist a large community of other researchers.
期刊论文(8)
专著(0)
科研奖励(0)
会议论文
登录
查看更多内容
DOI:
--
发表时间:
2018
期刊:
AAAI Conference on Human Computation
影响因子:
--
作者:
[C. Lin, Mausam]
通讯作者:
C. Lin, Mausam
Intelligible Artificial Intelligence
可理解的人工智能
DOI:
--
发表时间:
2018
期刊:
March 2018
影响因子:
--
作者:
[D.S. Weld, G. Bansal]
通讯作者:
D.S. Weld, G. Bansal
DOI:
10.18653/v1/n18-2058
发表时间:
2018-06
期刊:
ArXiv
影响因子:
--
作者:
[James Ferguson;Colin Lockard;Daniel S. Weld;Hannaneh Hajishirzi]
通讯作者:
James Ferguson;Colin Lockard;Daniel S. Weld;Hannaneh Hajishirzi
DOI:
10.1609/aaai.v32i1.11493
发表时间:
2018-04
期刊:
影响因子:
--
作者:
[Gagan Bansal;Daniel S. Weld]
通讯作者:
Gagan Bansal;Daniel S. Weld
DOI:
--
发表时间:
2018-11
期刊:
影响因子:
--
作者:
[Jonathan Bragg]
通讯作者:
Jonathan Bragg
共 7 条
CCRI: Research Infrastructure: NEW: Semantic Scholar Open Data Platform: Enabling Research Into Scientific Search and Discovery
-
批准号:2213656
-
项目类别:Standard Grant
-
资助金额:$200.0万
-
财政年份:2022
-
负责人:Daniel Weld
-
依托单位:
RAPID: Augmented Intelligence for Accelerating Covid-Related Scientific Discovery
-
批准号:2040196
-
项目类别:Standard Grant
-
资助金额:$20.0万
-
财政年份:2020
-
负责人:Daniel Weld
-
依托单位:
RI: Small: Decision-Theoretic Control of Crowd-Sourced Workflows
-
批准号:1016713
-
项目类别:Standard Grant
-
资助金额:$30.47万
-
财政年份:2010
-
负责人:Daniel Weld
-
依托单位:
RI: Small: Integrating Paradigms for Approximate Stochastic Planning
-
批准号:1016465
-
项目类别:Standard Grant
-
资助金额:$45.05万
-
财政年份:2010
-
负责人:Daniel Weld
-
依托单位:
Supporting Students Attending IUI 2009 Conference
-
批准号:0914591
-
项目类别:Standard Grant
-
资助金额:$1.44万
-
财政年份:2009
-
负责人:Daniel Weld
-
依托单位:
Representation and Reasoning about Adaptive Interfaces
-
批准号:0307906
-
项目类别:Continuing Grant
-
资助金额:$50.7万
-
财政年份:2003
-
负责人:Daniel Weld
-
依托单位:
Extending Graphplan to Handle Uncertainty and Sensing Actions
-
批准号:9872128
-
项目类别:Standard Grant
-
资助金额:$23.7万
-
财政年份:1998
-
负责人:Daniel Weld
-
依托单位:
Principled Planning with Simultaneous Actions, Metric Time and Continuous Effects
-
批准号:9303461
-
项目类别:Continuing Grant
-
资助金额:$40.0万
-
财政年份:1994
-
负责人:Daniel Weld
-
依托单位:
Presidential Young Investigator Award
-
批准号:8957302
-
项目类别:Continuing Grant
-
资助金额:$21.2万
-
财政年份:1989
-
负责人:Daniel Weld
-
依托单位:
Managing Complexity in Qualitative Physics
-
批准号:8902010
-
项目类别:Standard Grant
-
资助金额:$12.56万
-
财政年份:1989
-
负责人:Daniel Weld
-
依托单位:
国内基金
海外基金
登录
查看更多内容
昼夜节律性small RNA在血斑形成时间推断中的法医学应用研究
-
批准号:
-
项目类别:省市级项目
-
资助金额:--
-
批准年份:2024
-
负责人:
-
依托单位:
tRNA-derived small RNA上调YBX1/CCL5通路参与硼替佐米诱导慢性疼痛的机制研究
-
批准号:
-
项目类别:省市级项目
-
资助金额:10.0万元
-
批准年份:2022
-
负责人:张祥忠
-
依托单位:
Small RNA调控I-F型CRISPR-Cas适应性免疫性的应答及分子机制
-
批准号:32000033
-
项目类别:青年科学基金项目
-
资助金额:24.0万元
-
批准年份:2020
-
负责人:林平
-
依托单位:
Small RNAs调控解淀粉芽胞杆菌FZB42生防功能的机制研究
-
批准号:31972324
-
项目类别:面上项目
-
资助金额:58.0万元
-
批准年份:2019
-
负责人:高学文
-
依托单位:
变异链球菌small RNAs连接LuxS密度感应与生物膜形成的机制研究
-
批准号:81900988
-
项目类别:青年科学基金项目
-
资助金额:21.0万元
-
批准年份:2019
-
负责人:毛梦莹
-
依托单位:
肠道细菌关键small RNAs在克罗恩病发生发展中的功能和作用机制
-
批准号:31870821
-
项目类别:面上项目
-
资助金额:56.0万元
-
批准年份:2018
-
负责人:陈江宁
-
依托单位:
基于small RNA 测序技术解析鸽分泌鸽乳的分子机制
-
批准号:31802058
-
项目类别:青年科学基金项目
-
资助金额:26.0万元
-
批准年份:2018
-
负责人:麻慧
-
依托单位:
Small RNA介导的DNA甲基化调控的水稻草矮病毒致病机制
-
批准号:31772128
-
项目类别:面上项目
-
资助金额:60.0万元
-
批准年份:2017
-
负责人:吴建国
-
依托单位:
基于small RNA-seq的针灸治疗桥本甲状腺炎的免疫调控机制研究
-
批准号:81704176
-
项目类别:青年科学基金项目
-
资助金额:20.0万元
-
批准年份:2017
-
负责人:赵继梦
-
依托单位:
水稻OsSGS3与OsHEN1调控small RNAs合成及其对抗病性的调节
-
批准号:91640114
-
项目类别:重大研究计划
-
资助金额:85.0万元
-
批准年份:2016
-
负责人:何祖华
-
依托单位: