HCC: Small: Examining the Super User versus the Crowd in Human-Centered Computation

该项目研究了各种规模的基于人群的人类分析的性质，特别是少数贡献者的集中努力与许多贡献者的微观贡献之和有何不同。 自动化方法擅长处理大量数据，但在做出决策或观察时，它们缺乏人类感知的灵活性和敏感性，特别是当计算挑战围绕视觉分析时。作为替代方案，人类网络可以通过微任务的分布促进大规模并行计算来扩大人类感知，但人类数据的解释在个体之间是不同的。 在基于群体的计算中，个体参与量的广泛变化产生了群体的非统一表示，这是一个重要的差异，可能会显著影响“群体”一词在众包中的有效性。 该研究将探索从参与曲线的极端生成的数据，并量化从广泛的人群抽样与少数“超级用户”的集中声音生成的数据质量。作为一种比较措施，研究人员将观察人类生成分析的特征变量采样在用作机器学习框架中的训练数据时如何改变结果。 这项调查将利用众包工作产生的数据，该工作利用了1万多名志愿者参与，在超高分辨率卫星图像上生成了200多万个人工注释，以寻找蒙古各地的坟墓。图像瓦片随机分发给标记感兴趣异常的参与者，而人群对感兴趣点的共识为实地调查小组提供了蒙古地面实况的位置。参与者的范围很广，20%的数据来自最活跃的1%的参与者，而在另一个极端，20%的数据来自80%的参与者，他们最不活跃。 虽然人群的共识提供了一个衡量异常识别质量的指标，但地面实况观察显示，实际验证往往与更高兴趣参与者的识别相对应。 这项研究将探讨从专家与非专家群体产生的数据的性质，从参与水平的差异开始。最大的数据挑战。 众包的例子已经表明，分布式微任务的力量可以应对像星系分类这样巨大的挑战，或者像折叠蛋白质那样复杂的挑战。 然而，这一概念取决于招募人力帮助，通常是个人愿意做出贡献的任何参与水平。个人之间的贡献和影响水平的差异可能是惊人的，参与通常分布在长尾曲线上。在从生成的数据中提取知识时，应该认识和理解招募人群的基本方面。 该项目将通过确定群体的分布式输入与个人的集中努力有何不同，从而有助于必要的理解。 深入了解群体动态对结果的影响将决定我们如何汇集和保留参与，从而对众包作为分析概念的发展产生变革性影响。

This project investigates the nature of crowd-based human analytics at various scales, specifically how the concentrated efforts of a few contributors differ from the summed micro contributions of many. Automated approaches are good at handling huge amounts of data, but they lack the flexibility and sensitivity of human perception when making decisions or observations, especially when computational challenges revolve around visual analytics. Networks of humans, as an alternative, can scale up human perception by facilitating massively parallel computation through the distribution of micro-tasks, but human data interpretation is variant between individuals. Wide variability in the amount of participation of individuals in crowd-based computation creates non-uniform representations of a crowd, which is an important discrepancy that could significantly impact the validity of the term "crowd" in crowdsourcing. The research will explore data generated from the extreme ends of the participation curve and quantify the quality of data produced from a broad sampling of a crowd versus concentrated voice of the few "super users." As one measure of comparison, the researchers will observe how characteristically variant samplings of human generated analysis alter the outcome when used as training data in a machine learning framework. This investigation will utilize data generated from a crowdsourcing effort that tapped over 10,000 volunteer participants to generate over 2 million human annotations on ultra-high resolution satellite imagery in search for tombs across Mongolia. Image tiles were distributed at random to participants who tagged anomalies of interest, while crowd consensus on points of interest provided a field survey team with locations to ground truth in Mongolia. Participation ranged widely, as illustrated by the fact that 20 percent of the data came from the most active 1 percent of participants, while at the other extreme 20 percent of the data came from the 80 percent of participants who were least active. While consensus of the crowd provided one metric to measure the quality of anomaly identifications, ground truth observations showed actual validation tended to correspond with identifications made from higher interest participants. This study will explore the nature of data generated from experts versus crowds of non-experts, starting from the discrepancies in participation levels.Crowd-based human analytics has been welcomed as a potential solution to some of the world?s largest data challenges. Examples of crowdsourcing have shown that the power of distributed microtasking can engage challenges as overwhelming as categorizing the galaxies, or as complicated as folding proteins. However this concept depends upon the recruitment of human help, often at whatever levels of participation an individual is willing to contribute. The variation in contributions, and thus impact levels, between individuals can be staggering, with participation typically distributed across a longtail curve. That fundamental aspect of a recruited crowd should be recognized and understood when extracting knowledge from the data that is generated. This project will contribute to the necessary understanding by determining how the distributed inputs from a crowd differ from the concentrated efforts of an individual. Insight into the effects of crowd dynamics on results will determine how we pool and retain participation and, thus, have transformative impact on the development of crowdsourcing as a concept for analytics.