CAREER: Continual Automated Refinement of Human Computation Systems

这项研究的目的是改进自动化工具，用于人类和计算问题解决系统的应用。这将导致数据驱动建模和优化设计此类系统过程的通用技术，减少创建成功系统所需的工作量，并扩大可以应用大量人力资源的问题域的范围。尽管目前拥有巨大的计算能力，但许多重要问题仍然依赖于人类的推理或直觉来解决。在算法未知或计算难以处理的情况下，人类计算最近作为一种应用人类技能来推进解决人类和计算机都无法单独解决的问题的手段而出现。通过将人类的创造力、解决问题的能力和视角发挥出来，人类和计算机结合起来可以解决以前无法解决的问题。此外，这些系统为参与科学创造了一条新的途径——一种人们为解决对他们重要的问题作出贡献的新途径。通过科学的民主化，我们让那些原本没有这种手段的人参与进来。最后，这项研究有助于我们理解如何最好地训练人们解决具有挑战性的问题。这项工作旨在自动化人类计算系统迭代改进的一个方面：改进对贡献者的任务分配。基本方法是基于技能等级和技能链构建贡献者和任务的模型，该模型可用于为贡献者分配要完成的适当任务。该模型将根据数据自动改进技能评估和分配，从而改善用户体验和解决问题的结果。这种方法可以分解为三个挑战领域：1)开发一个结合技能原子和技能等级的统一技能模型，然后使用该技能模型；2)为每个参与者量身定制一个难度曲线；3)评估设计决策。该方法将建立在现有的多人配对系统上，并在多个人类计算系统中进行验证。

This research aims to improve automated tools for the application of human and computational problem-solving systems. This will lead to generalized techniques for data-driven modeling and optimization of the process of designing such systems, reducing the workload necessary to create successful ones and broadening the scope of problem domains to which massive amounts of human brainpower can be applied. Despite the vast computational power currently available, a broad range of important problems still rely on human reasoning or intuition to solve. In cases where algorithms are either unknown or computationally intractable, human computation has recently arisen as a means to apply human skills to advance solutions to problems neither humans nor computers could solve alone. By bringing human creativity, problem solving, and perspective to bear, humans and computers combined can solve previously unsolvable problems. Additionally, these systems create a new pathway for involvement in science - a new way for people to contribute towards problems that are important to them. By democratizing science, we involve those who may not otherwise have had such a means. Finally, this research can contribute to our understanding of how to best train people in solving challenging problems. This work seeks to automate one aspect of the iterative refinement of human computation systems: improving the assignment of tasks to contributors. The basic approach is to construct a model of contributors and tasks, based on skill ratings and skill chains, which can be used to assign contributors an appropriate task to complete. This model will automatically refine the skill estimates and assignments over time based on data, improving both user experience and problem solving outcomes. This approach in broken down into three challenge areas: 1) developing a unified skill model that combines skill atoms and skill ratings, then using that skill model for 2) crafting a difficulty curve tailored for each participant, and 3) evaluating design decisions. The approach will build on existing multi-person matchmaking systems, validated in multiple human computation systems.