CAREER: Continual Automated Refinement of Human Computation Systems

本研究旨在改进人类和计算问题解决系统应用的自动化工具。这将导致数据驱动建模和优化设计此类系统的过程的通用技术，减少创建成功系统所需的工作量，并扩大可应用大量人类脑力的问题领域的范围。尽管目前有巨大的计算能力，但许多重要问题仍然依赖于人类的推理或直觉来解决。在算法未知或计算困难的情况下，人类计算最近出现了一种应用人类技能来推进人类或计算机都无法单独解决的问题的解决方案的手段。通过发挥人类的创造力、解决问题的能力和洞察力，人类和计算机的结合可以解决以前无法解决的问题。此外，这些系统为参与科学创造了一条新的途径--一种让人们为解决对他们来说重要的问题做出贡献的新方式。通过使科学民主化，我们让那些本来没有这种手段的人参与进来。最后，这项研究有助于我们了解如何最好地训练人们解决具有挑战性的问题。这项工作旨在自动化人类计算系统的迭代改进的一个方面：改进任务分配给贡献者。基本方法是基于技能评级和技能链构建贡献者和任务的模型，该模型可用于为贡献者分配要完成的适当任务。该模型将根据数据自动优化技能估计和分配，从而改善用户体验和解决问题的结果。这种方法分为三个挑战领域：1）开发一个统一的技能模型，结合技能原子和技能评级，然后使用该技能模型2）为每个参与者量身定制难度曲线，以及3）评估设计决策。该方法将建立在现有的多人匹配系统上，并在多个人类计算系统中得到验证。

This research aims to improve automated tools for the application of human and computational problem-solving systems. This will lead to generalized techniques for data-driven modeling and optimization of the process of designing such systems, reducing the workload necessary to create successful ones and broadening the scope of problem domains to which massive amounts of human brainpower can be applied. Despite the vast computational power currently available, a broad range of important problems still rely on human reasoning or intuition to solve. In cases where algorithms are either unknown or computationally intractable, human computation has recently arisen as a means to apply human skills to advance solutions to problems neither humans nor computers could solve alone. By bringing human creativity, problem solving, and perspective to bear, humans and computers combined can solve previously unsolvable problems. Additionally, these systems create a new pathway for involvement in science - a new way for people to contribute towards problems that are important to them. By democratizing science, we involve those who may not otherwise have had such a means. Finally, this research can contribute to our understanding of how to best train people in solving challenging problems. This work seeks to automate one aspect of the iterative refinement of human computation systems: improving the assignment of tasks to contributors. The basic approach is to construct a model of contributors and tasks, based on skill ratings and skill chains, which can be used to assign contributors an appropriate task to complete. This model will automatically refine the skill estimates and assignments over time based on data, improving both user experience and problem solving outcomes. This approach in broken down into three challenge areas: 1) developing a unified skill model that combines skill atoms and skill ratings, then using that skill model for 2) crafting a difficulty curve tailored for each participant, and 3) evaluating design decisions. The approach will build on existing multi-person matchmaking systems, validated in multiple human computation systems.