EAGER: Human Computation: Integrating the Crowd and the Machine

EAGER：人类计算：整合人群和机器

• 批准号: 1145291
• 负责人: Albert Lin
• 金额: $ 6.6万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-08-01 至 2013-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1145291&HistoricalAwards=false
• 关键词: EAGER; Human; Computation; Integrating; Crowd

Because both information and connectivity are more available today than ever before thanks to digital technologies, questions can now be addressed by enlisting massive human demographics to supplement the limitations of computer computation. This is especially relevant in the case of visual analytics, where human intuition remains far superior to existing computer object recognition algorithms. While algorithms are limited by pre-labeling requirements, humans can perceive subtle variations and nuances to identify and classify unexpected objects. These tasks, however, are often too massive in scale for a single human to accomplish. Distributing this task over a massive network not only succeeds in categorizing data, but generates massive quantities of human quantifiers (training data) to potentially teach computer vision algorithms to mimic human perception in order to distinguish the normal from the abnormal.This exploratory project will combine collective human visual perception with machine learning and object recognition, through a study of 1.25 million crowd-sourced inputs provided by over 6,000 volunteers labeling satellite imagery in a search for anomalies in northern Mongolia. These data, collected from June 2010 to the present via an online platform developed by the PI in collaboration with National Geographic Digital Media, afford an ideal "case study" environment to investigate the nature of crowd generated data and methods that distill the wide variability of human input into computational algorithms. The online participants, excited by the potential of discovering the tomb of Genghis Khan, examined massive amounts of ultra-high resolution multispectral satellite imagery to label loosely defined anomalies into various categories. Trends that emerged from the massive volume of labels represent a collective human perspective on what the images contain. A team led by the PI traveled to Mongolia to ground-truth areas of high user input convergence. The resulting ground-truthed anomalies provide a unique opportunity to both accurately measure the quality of human/automated analysis and to investigate the effect of supplementing noisy crowd-sourced data sets with small pools of absolute data in machine learning. In the current project the PI will develop a framework for applying and evaluating the following three research phases designed to study the nature of large scale human generated data for integration into supervised learning algorithms:1. Consensus Clustering - Tag evaluation mechanisms based upon the volume and consistency of neighboring tags and the ability of the individuals creating those tags. Unsupervised methods for "merging" labels will also be applied for extended anomalies such as roads and rivers.2. Feature Vector Extraction - Both the type of features (e.g., color, luminance, edges and gradients, scale, orientation, etc.) and the extent of the neighborhoods (e.g., local, wide and global) required to detect anomalies are unknown a priori. Thus, the aim is to determine sufficiently diverse features to capture all relevant cues within the image.3. Machine Learning - Dominant features representative of, and excluded from, pixel groups of given categories will be determined from the results of Phase 2 above.Broader Impacts: In this exploratory study the PI will lay the foundation for extracting new machine/human collaborative opportunities from the resource of the crowd. Understanding the bonds between human and computer intelligence will have a profound impact on many branches of science. Thus, concepts developed in this effort may ultimately prove transformative by affording migration of crowd-sourcing from a project-based tool for distributed analytics into a portal bridging collective human perception and machine learning.

由于今天的信息和连接比以往任何时候都更容易获得，这要归功于数字技术，现在可以通过招募大量的人口统计学数据来补充计算机计算的局限性来解决问题。这一点在视觉分析的情况下尤其相关，人类的直觉仍然远远优于现有的计算机对象识别算法。虽然算法受到预先标记要求的限制，但人类可以感知到细微的变化和细微差别，以识别和分类意想不到的对象。然而，这些任务往往规模太大，一个人无法完成。将这项任务分布在一个大规模的网络上，不仅成功地对数据进行分类，而且生成了大量的人类量词(训练数据)，潜在地教导计算机视觉算法模仿人类的感知来区分正常和异常。这个探索性项目将把人类的集体视觉感知与机器学习和目标识别结合起来，通过研究6000多名志愿者提供的125万份众包输入来寻找蒙古北部的异常情况，这些志愿者标记了卫星图像。这些数据是从2010年6月至今通过PI与国家地理数字媒体合作开发的在线平台收集的，为调查人群生成数据的性质和将人类输入的广泛差异提取到计算算法中的方法提供了一个理想的“案例研究”环境。在线参与者对发现成吉思汗陵墓的潜力感到兴奋，他们检查了大量超高分辨率的多光谱卫星图像，将松散定义的异常划分为各种类别。从大量标签中出现的趋势代表了人类对图像所包含内容的集体视角。由PI领导的一个小组前往蒙古，前往用户输入趋同程度较高的实地真相地区。由此产生的地面真实异常提供了一个独特的机会，既可以准确衡量人工/自动分析的质量，也可以调查在机器学习中用小的绝对数据池补充嘈杂的众包数据集的效果。在目前的项目中，PI将开发一个框架，用于应用和评估以下三个研究阶段，这些研究阶段旨在研究大规模人类生成数据的性质，以便集成到监督学习算法中：1.共识聚类-标签评估机制基于相邻标签的数量和一致性以及创建这些标签的个人的能力。用于“合并”标签的非监督方法也将被应用于扩展异常，例如道路和河流。特征向量提取-包括特征类型(例如，颜色、亮度、边缘和渐变、比例、方向等)并且检测异常所需的邻域(例如，局部、广泛和全局)的范围是先验未知的。因此，目标是确定足够多样化的特征，以捕捉图像中所有相关的线索。机器学习-代表给定类别的像素组以及排除在其之外的主要特征将从上述第二阶段的结果中确定。广泛影响：在这项探索性研究中，PI将为从人群资源中提取新的机器/人类协作机会奠定基础。了解人类和计算机智能之间的联系将对许多科学分支产生深远的影响。因此，在这一努力中提出的概念可能最终被证明是变革性的，因为它提供了从基于项目的分布式分析工具到连接人类集体感知和机器学习的门户的迁移。