HCC: Small: Examining the Super User versus the Crowd in Human-Centered Computation

这个项目在不同的尺度上调查了基于人群的人类分析的性质，特别是少数贡献者的集中努力与许多人的总体微观贡献有何不同。自动化方法擅长处理海量数据，但它们在做出决策或观察时缺乏人类感知的灵活性和敏感度，特别是当计算挑战围绕视觉分析展开时。作为另一种选择，人类网络可以通过分配微任务来促进大规模并行计算，从而扩大人类的感知能力，但人类数据的解释因个人而异。在基于群体的计算中，个人参与量的广泛差异造成了对群体的非统一表示，这是一个重要的差异，可能会显著影响众包中“群体”一词的有效性。这项研究将探索从参与曲线的两端产生的数据，并量化从人群的广泛抽样产生的数据质量，而不是少数“超级用户”的集中声音。作为比较的一种方法，研究人员将观察在机器学习框架中用作训练数据时，人类产生的分析的特征变异样本如何改变结果。这项调查将利用众包工作产生的数据，利用1万多名志愿者参与，在超高分辨率卫星图像上生成200多万条人类注释，在蒙古各地寻找坟墓。图像瓷砖被随机分发给参与者，他们标记出感兴趣的异常情况，而人群对兴趣点的共识为实地调查小组提供了在蒙古找到真相的地点。参与者的范围很广，这一事实表明，20%的数据来自最活跃的1%的参与者，而在另一个极端，20%的数据来自最不活跃的80%的参与者。虽然人群的共识提供了一个衡量异常识别质量的指标，但地面事实观察显示，实际验证往往与兴趣较高的参与者的识别相一致。这项研究将从参与程度的差异入手，探索专家和非专家群体产生的数据的性质。基于冠的人类分析作为一种潜在的解决方案受到了欢迎，这是世界上一些最大的数据挑战？S。众包的例子表明，分布式微任务的力量可能会遇到像对星系进行分类一样压倒性的挑战，或者像折叠蛋白质一样复杂。然而，这一概念依赖于招募人类的帮助，通常是在个人愿意贡献的参与程度上。个人之间的贡献和影响程度的差异可能是惊人的，参与通常分布在一条长尾曲线上。在从产生的数据中提取知识时，应认识和理解招募人群的这一基本方面。这个项目将通过确定来自人群的分散投入与个人的集中努力有何不同，从而有助于必要的理解。洞察人群动态对结果的影响将决定我们如何汇集和保持参与，从而对众包作为一个分析概念的发展产生变革性的影响。

This project investigates the nature of crowd-based human analytics at various scales, specifically how the concentrated efforts of a few contributors differ from the summed micro contributions of many. Automated approaches are good at handling huge amounts of data, but they lack the flexibility and sensitivity of human perception when making decisions or observations, especially when computational challenges revolve around visual analytics. Networks of humans, as an alternative, can scale up human perception by facilitating massively parallel computation through the distribution of micro-tasks, but human data interpretation is variant between individuals. Wide variability in the amount of participation of individuals in crowd-based computation creates non-uniform representations of a crowd, which is an important discrepancy that could significantly impact the validity of the term "crowd" in crowdsourcing. The research will explore data generated from the extreme ends of the participation curve and quantify the quality of data produced from a broad sampling of a crowd versus concentrated voice of the few "super users." As one measure of comparison, the researchers will observe how characteristically variant samplings of human generated analysis alter the outcome when used as training data in a machine learning framework. This investigation will utilize data generated from a crowdsourcing effort that tapped over 10,000 volunteer participants to generate over 2 million human annotations on ultra-high resolution satellite imagery in search for tombs across Mongolia. Image tiles were distributed at random to participants who tagged anomalies of interest, while crowd consensus on points of interest provided a field survey team with locations to ground truth in Mongolia. Participation ranged widely, as illustrated by the fact that 20 percent of the data came from the most active 1 percent of participants, while at the other extreme 20 percent of the data came from the 80 percent of participants who were least active. While consensus of the crowd provided one metric to measure the quality of anomaly identifications, ground truth observations showed actual validation tended to correspond with identifications made from higher interest participants. This study will explore the nature of data generated from experts versus crowds of non-experts, starting from the discrepancies in participation levels.Crowd-based human analytics has been welcomed as a potential solution to some of the world?s largest data challenges. Examples of crowdsourcing have shown that the power of distributed microtasking can engage challenges as overwhelming as categorizing the galaxies, or as complicated as folding proteins. However this concept depends upon the recruitment of human help, often at whatever levels of participation an individual is willing to contribute. The variation in contributions, and thus impact levels, between individuals can be staggering, with participation typically distributed across a longtail curve. That fundamental aspect of a recruited crowd should be recognized and understood when extracting knowledge from the data that is generated. This project will contribute to the necessary understanding by determining how the distributed inputs from a crowd differ from the concentrated efforts of an individual. Insight into the effects of crowd dynamics on results will determine how we pool and retain participation and, thus, have transformative impact on the development of crowdsourcing as a concept for analytics.