RI: Small: Decision-Theoretic Control of Crowd-Sourced Workflows

RI：小型：众包工作流程的决策理论控制

• 批准号: 1016713
• 负责人: Daniel Weld
• 金额: $ 30.47万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-09-15 至 2014-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1016713&HistoricalAwards=false
• 关键词: RI; Small; Decision; Theoretic; Control

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people as an open request for services. Requesters use crowd-sourcing for a wide variety of jobs like dictation-transcription, content screening, linguistic tasks, user-studies, etc. These requesters often use complex workflows to subdivide a large task into bite-sized pieces (including the management of these tasks), each of which is independently crowd-sourced. These workflows are paramount to the success of crowd-sourcing, still, there has been little attention paid to methods for dynamically optimizing the throughput of a workflow. Controlling and optimizing such a workflow is an excellent application for AI research for two reasons. First, it is challenging in that the agent has to understand the dynamics of an uncertain, real-time environment and reason about distinct choices for a decision. More importantly, the domain has significant economic value -- progress can potentially impact hundreds of thousands of people and spur economic development in a fast growing sector.This project is investigating complex workflows using a decision-theoretic framework that optimizes for a quality/price trade-off, with aims of (1) building statistical models of worker behavior derived from a large corpus of online behavior, (2) defining a declarative representation language to describe a wide range of workflows, and (3) developing an automated scheme that optimizes a general workflow resulting in an automated controller for making informed decisions at various stages of the process and for monitoring worker accuracies and computing corrections based on them. In the longer term, perhaps beyond the scope of this project, is (4) development of an interface optimizer that automatically learns the best user interface for a task based on user behavior increasing throughput of the workflow, and (5) integrating these ideas in an open-source, software toolkit to directly benefit the various requesters in managing their tasks.

众包（crowd -sourcing）是最近出现的一种框架，在这种框架中，人类智能任务被外包给一群不知名的人，作为对服务的公开请求。请求者将众包用于各种各样的工作，如口述转录、内容筛选、语言任务、用户研究等。这些请求者通常使用复杂的工作流将大型任务细分为小块（包括这些任务的管理），每个任务都是独立的众包。这些工作流对于众包的成功至关重要，然而，很少有人关注动态优化工作流吞吐量的方法。控制和优化这样的工作流是人工智能研究的一个很好的应用，原因有两个。首先，它具有挑战性，因为代理必须理解不确定的实时环境的动态，并对决策的不同选择进行推理。更重要的是，该领域具有显著的经济价值——其进步可能会影响成千上万的人，并在一个快速增长的领域刺激经济发展。该项目使用优化质量/价格权衡的决策理论框架来研究复杂的工作流，其目的是(1)从大量在线行为语料库中构建工人行为的统计模型，(2)定义一种声明性表示语言来描述广泛的工作流，(3)开发一种自动化方案，该方案优化一般工作流程，从而产生一种自动控制器，用于在过程的各个阶段做出明智的决策，并用于监控工人的准确性并基于它们计算更正。从长远来看，也许超出了这个项目的范围，是(4)开发一个界面优化器，它可以根据用户行为自动学习任务的最佳用户界面，从而提高工作流的吞吐量，以及(5)将这些想法集成到一个开源的软件工具包中，以直接使各种请求者在管理他们的任务时受益。