Multimodal Disaster Impact Assessment Models for Enhanced Resilience

在发生重大灾害事件(如野火、地震、洪水)后，立即对基础设施受损的地理分布和严重程度进行评估，这对应急响应和早期恢复计划的成功至关重要。这种情况意识是由设施所有者、用户、应急人员以及地方和州官员实施的决策过程的重要组成部分。相反，对建筑环境受影响状态普遍缺乏了解，可能会导致公众反应混乱，恢复速度变慢。虽然可以通过建筑专业人员进行的现场检查来全面评估基础设施损坏的程度和分布，但这可能是一个漫长的、资源密集型的过程，具体取决于活动的规模。该灾难恢复研究补助金(DRRG)项目将通过利用人工智能(AI)的原理来开发能够处理和利用不同类型的数据和信息(例如，图像、文本、表格数据)的近乎实时的基础设施损坏预测模型来应对这一挑战。通过提高我们有效整合不同信息源的能力，该项目旨在改变在重大灾难事件发生后对基础设施的有形损害进行评估的方式，从而加强随后的应急和恢复规划阶段。这项研究将为博士生和硕士学生提供培训，并有机会向来自不同背景的本科生传授如何将科学和工程与人工智能技术结合起来，以改善社区对极端事件的反应。多模式学习中的基本概念和方法进步将用于改造和增强基础设施损坏预测模型，以便在紧随其后的事件环境中使用。基于图像的损伤评估将沿着两个维度发展：(1)发展基于视觉变换的方法；(2)建立一种使用大量未标记数据集来训练模型的自监督学习方法。围绕基础设施损坏预测模型多通道数据融合的广阔领域，将建立一个新的知识库。将回答关于统一表示、跨模式和数据融合之间的转换和对齐的具体问题。这项研究还将产生一种新型的与灾害无关的基础设施损坏预测模型。这种模型将能够接收一种或多种类型的输入模式(即，图像、文本和工程/表格数据)的综合表示，并产生与因果事件的类型(例如，地震或飓风)无关的基础设施损坏程度作为输出。使用来自多个自然灾害事件(飓风和地震)的综合数据集，该项目将包括实验，以揭示特定于灾害和与灾害无关的多模式模型增强早期基础设施损害评估的能力，以提高对情况的感知和增强的复原力。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Immediately after a major hazard event (e.g., wildfire, earthquake, flood), a prompt assessment of the geographic distribution and severity of infrastructure damage is vital to the success of the emergency response and early recovery planning. This situational awareness is an important part of the decision-making processes that are implemented by facility owners, users, emergency responders and local and state officials. Conversely, a general lack of knowledge about the impacted state of the built environment can lead to a disorganized public response and slower recovery. While a comprehensive assessment of the extent and distribution of infrastructure damage can be obtained from in-person inspections conducted by building professionals, depending on the scale of the event, this can be a lengthy, resource intensive process. This Disaster Resilience Research Grants (DRRG) project will address this challenge by utilizing principles from artificial intelligence (AI) to develop near real-time infrastructure damage prediction models that can process and utilize different types of data and information (e.g., images, text, tabular data). By advancing our ability to effectively integrate disparate information sources, this project aims to transform the way that physical damage to infrastructure is estimated in the aftermath of a major disaster event, thereby enhancing the emergency response and recovery planning phases that follow. The research will provide training for doctoral and masters students and an opportunity to teach undergraduates from different backgrounds how science and engineering coupled with AI technologies can be used to improve community response to extreme events.Fundamental concepts and methodological advancements in multimodal learning will be used to transform and enhance infrastructure damage prediction models for use in the immediate post-event environment. The state-of-the-art in image-based damage assessment will be advanced along two dimensions: (1) developing Vision Transformer-based methods and (2) establishing a self-supervised learning methodology for training the models using large collections of unlabeled data. A new knowledge base will be established around the broad area of multimodal data fusion for infrastructure damage prediction models. Specific questions regarding unified representation, translation across and alignment between modalities and data fusion will be answered. A new type of hazard-agnostic infrastructure damage prediction model will also emerge from this research. Such a model will have the ability to receive an integrated representation of one or more types of input modalities (i.e., image, text, and engineering/tabular data) and produce, as output, an infrastructure damage level that is agnostic to the type of causal event (e.g., earthquake or hurricane). Using a comprehensive data set from multiple natural hazard events (hurricane and earthquake), the project will include experiments to shed new light on the ability of both hazard-specific and hazard-agnostic multimodal models to enhance early-stage infrastructure damage assessments for increased situational awareness and enhanced resilience.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.