CAREER: Advancing Open-Ended Crowdsourcing: The Next Frontier in Crowdsourced Data Management

基于大数据的机器学习终于对我们的日常生活产生了影响，从Siri和谷歌翻译这样的小成功，到无人驾驶汽车和计算机辅助医学图像诊断等更艰难的新兴应用。 从普通的在线欺诈检测到最复杂的计算机视觉应用，这些应用都对大量标记的训练数据有着无法满足的需求。高质量标签的主要来源是众包，迄今为止关于众包的研究集中在如何最大限度地提高每美元生产高质量众包标签的关键问题上，对于工人必须在几个预定义的标签之间进行选择的问题。 然而，越来越多的开放式标签问题已经发展到构成了几乎一半的众包任务，开放式任务提出了一系列全新的研究挑战，众包数据管理。这项活动解决了管理和优化开放式众包的关键新的研究挑战。由于开放式众包采用的任务有大量的替代品，人类很难选择没有错误的任务。在确定适合特定问题的开放式任务类型，制定计划以确定开放式工人响应的正确答案，以及推断工人答案背后隐藏的观点方面出现了其他挑战。该活动针对的是开放式众包问题，涵盖了当今实践中近90%的问题，在计算机视觉、自然语言处理和机器学习中具有广泛的适用性。该活动的技术成果包括开放式众包数据管理的第一个基本原则，这反过来将扩大机器学习的范围，使其进入新的更具挑战性的领域，并在影响我们日常生活的现有应用中提供更有效的解决方案。该活动的教学成果包括：关于人在环数据分析的课程、学校教师众包教育模块，以及对实践中如何进行众包的量化和传播，沿着一个基准，以加快今后的众包研究。

Machine learning on big data is finally having an impact on our daily lives, from small triumphs like Siri and Google Translate to much tougher emerging applications like driverless cars and computer-assisted medical image diagnosis. From mundane online fraud detection to the most sophisticated uses of computer vision, these applications share an insatiable appetite for massive labeled training data. The primary source of high-quality labels is crowdsourcing, and research to date on crowdsourcing has focused on the key problem of how to maximize the production of high-quality crowdsourced labels per dollar spent, for problems where workers must choose between just a few predefined labels. However, more open-ended labeling problems have grown to constitute almost half of crowdsourced tasks today, and open-ended tasks raise an entirely new set of research challenges for crowdsourced data management.This activity addresses the key new research challenges in managing and optimizing open-ended crowdsourcing. Since open-ended crowdsourcing employs tasks with a large number of alternatives, humans struggle to select error-free ones. Additional challenges emerge in determining the open-ended task types appropriate for a specific problem, developing schemes to ascertain the right answer given open-ended worker responses, and inferring the hidden perspectives behind worker answers. The activity targets open-ended crowdsourcing problems that span nearly 90% of those used in practice today, with wide applicability in computer vision, natural language processing, and machine learning in general. The technical outcomes of the activity include the first foundational principles for open-ended crowdsourced data management, which in turn will expand the reach of machine learning into new and more challenging domains and more effective solutions in existing applications that impact our everyday lives. The pedagogical outcomes of the activity include a course on human-in-the-loop data analytics, crowdsourcing education modules for school teachers, as well as a quantification and dissemination of how crowdsourcing is performed in practice, along with a benchmark to accelerate crowdsourcing research in the future.