RI: Small: Decision-Theoretic Control of Crowd-Sourced Workflows

众包是一个最近的框架，其中人类智能任务外包给一群未知的人，作为一个开放的服务请求。这些请求者通常使用复杂的工作流程将大型任务细分为小块（包括这些任务的管理），每个任务都是独立的众包。这些工作流是至关重要的众包的成功，仍然，很少有人关注动态优化工作流的吞吐量的方法。控制和优化这样的工作流程是人工智能研究的一个很好的应用，原因有两个。首先，它是具有挑战性的，因为代理人必须了解一个不确定的，实时的环境和原因的不同选择的决定的动态。更重要的是，该领域具有重要的经济价值--进步可能会影响数十万人，并刺激快速增长部门的经济发展。本项目使用优化质量/价格权衡的决策理论框架来研究复杂的工作流程，其目标是（1）从大量在线行为中构建工人行为的统计模型，（2）定义声明性表示语言以描述广泛的工作流，以及（3）开发优化一般工作流的自动化方案，从而产生自动化控制器，用于在过程的各个阶段做出明智的决策，并且用于监视工人的准确性并基于它们计算校正。从长远来看，也许超出了这个项目的范围，是（4）开发一个界面优化器，它可以根据用户行为自动学习任务的最佳用户界面，从而增加工作流的吞吐量，以及（5）将这些想法集成到开源软件工具包中，以直接使各种请求者在管理他们的任务时受益。

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people as an open request for services. Requesters use crowd-sourcing for a wide variety of jobs like dictation-transcription, content screening, linguistic tasks, user-studies, etc. These requesters often use complex workflows to subdivide a large task into bite-sized pieces (including the management of these tasks), each of which is independently crowd-sourced. These workflows are paramount to the success of crowd-sourcing, still, there has been little attention paid to methods for dynamically optimizing the throughput of a workflow. Controlling and optimizing such a workflow is an excellent application for AI research for two reasons. First, it is challenging in that the agent has to understand the dynamics of an uncertain, real-time environment and reason about distinct choices for a decision. More importantly, the domain has significant economic value -- progress can potentially impact hundreds of thousands of people and spur economic development in a fast growing sector.This project is investigating complex workflows using a decision-theoretic framework that optimizes for a quality/price trade-off, with aims of (1) building statistical models of worker behavior derived from a large corpus of online behavior, (2) defining a declarative representation language to describe a wide range of workflows, and (3) developing an automated scheme that optimizes a general workflow resulting in an automated controller for making informed decisions at various stages of the process and for monitoring worker accuracies and computing corrections based on them. In the longer term, perhaps beyond the scope of this project, is (4) development of an interface optimizer that automatically learns the best user interface for a task based on user behavior increasing throughput of the workflow, and (5) integrating these ideas in an open-source, software toolkit to directly benefit the various requesters in managing their tasks.