Citizen Science EAGER: Quantifying Uncertainty in Crowd Response for Reliable Wind Hazard and Damage Assessment

在美国，风暴对基础设施造成的破坏超过了任何其他自然灾害造成的破坏。然而，风暴期间建筑物上的风荷载具有高度可变的性质，这意味着当前的测量网络可能无法捕捉到损坏位置的准确特征。无处不在的智能手机和互联网，社交媒体的广泛使用和快速传播，参与科学努力的人群的力量，以及公众对漏洞的认识，都表明总体上感知危险的范式发生了转变。在风暴破坏的情况下，由公民科学公众参与检索和共享的实地数据可能会提供以前无法获得的风暴数据。这个早期概念探索性研究(AGERGE)奖的主要焦点是研究通过Amazon Machine Turk收集的“人类传感器”数据--这是一个众包应用程序。志愿者将被展示来自实际风暴的图像和相关数据，并被要求描述损害的特征和他们对评估的信心。这些数据将被用来设计一种众包算法，使公民科学的公众能够积极参与快速识别受损区域，帮助决策者分配资源，用于破坏响应和恢复工作，以及有针对性的破坏评估，这可以帮助改进易受强烈风暴影响的地区的建筑设计。这个项目将解决两个关键问题：如何量化众包损失评估的信心？在参与者不可靠的情况下，如何才能设计出更可靠的众包工具？研究人员最初将汇编一组数据，包括受密苏里州乔普林龙卷风影响的大约8000座建筑的图像、受损状态和风速估计。该数据集将用于以在线问卷的形式为公众创建可靠的众包图像分类方案。问卷将通过收集亚马逊机械土耳其上参与者的评估报告进行测试。这些报告将为开发参与者可靠性的不确定性模型提供信息，该模型构成了编码理论众包算法的基础，该算法对由于不可靠的参与者而产生的不确定性具有健壮性。该算法将在一个单独的数据集上进行测试，以比较研究人员的方法和没有控制参与者不可靠性的方法。最终的研究结果将与美国国家海洋和大气局分享，用于培训测量员，以评估风灾并为公众提供教程。

Damage to infrastructure arising from windstorms exceeds damage from any other natural hazard in the U.S. The highly variable nature of wind loadings on buildings during a windstorm, however, means that accurate characterization at the damage location may not be captured by current measurement networks. Ubiquitous smartphone and internet availability, widespread use and rapid dissemination of social media, the power of crowds engaged in scientific endeavors, and the public's awareness of vulnerabilities point to a paradigm shift in sensing hazards in general. In the case of windstorm damage, on-the-ground data retrieved and shared by Citizen Science public participation may provide windstorm data previously unavailable. The primary focus of this EArly-concept Grant for Exploratory Research (EAGER) award is to study "human-sensor" data collected through Amazon Mechanical Turk-- a crowd sourcing application. Volunteers will be shown images and related data from actual windstorms and asked to characterize the damage and their confidence in their assessments. These data will be used to design a crowd sourcing algorithm that will enable robust Citizen Science public participation in the rapid identification of damage areas to help decision makers to allocate resources for damage response and recovery efforts and for targeted damage assessments, which can help to improve the design of buildings in regions susceptible to intense windstorms. This project will address two key questions: How can one quantify the confidence in crowdsourced damage assessment? How can one design a tool for more reliable crowdsourcing given unreliable participants? Researchers will initially compile a "validation set" of data that includes imagery, damage states and wind speed estimates for approximately 8,000 structures that were affected by the Joplin, MO tornado. The data set will be used to create a reliable crowdsourced image classification scheme in the form of online questionnaires for the public. The questionnaires will be tested by collecting assessment reports from participants on Amazon Mechanical Turk. These reports will inform development of an uncertainty model for participant reliability, which forms the basis for a coding-theoretic crowdsourcing algorithm that is robust to uncertainties due to unreliable participants. This algorithm will be tested against a separate dataset to compare the researchers' approach with one that doesn't control for participant unreliability. The final research results will be shared with NOAA for use in training surveyors to assess wind damage and to provide tutorials for the public.