Multimodal Disaster Impact Assessment Models for Enhanced Resilience

重大危险事件发生后（例如，野火、地震、洪水），迅速评估基础设施破坏的地理分布和严重程度对于应急反应和早期恢复规划的成功至关重要。这种态势感知是设施所有者、用户、应急人员以及地方和州官员实施的决策过程的重要组成部分。相反，对建筑环境的受影响状况普遍缺乏了解，可能导致公众反应混乱，恢复缓慢。虽然可以通过建筑专业人员进行的现场检查对基础设施损坏的程度和分布情况进行全面评估，但这取决于事件的规模，这可能是一个漫长的资源密集型过程。这个灾难恢复研究赠款（DRRG）项目将通过利用人工智能（AI）的原理来解决这一挑战，以开发近实时的基础设施损坏预测模型，该模型可以处理和利用不同类型的数据和信息（例如，图像、文本、表格数据）。通过提高我们有效整合不同信息来源的能力，该项目旨在改变在重大灾害事件发生后估计基础设施实际损失的方式，从而加强随后的应急反应和恢复规划阶段。该研究将为博士生和硕士生提供培训，并为来自不同背景的本科生提供机会，让他们了解如何将科学和工程技术与人工智能技术相结合，以改善社区对极端事件的反应。多模式学习的基本概念和方法进步将用于改造和增强基础设施损坏预测模型，以用于事件发生后的即时环境。基于图像的损伤评估的最新技术将沿着沿着两个维度推进：（1）开发基于视觉变换器的方法和（2）建立一种自监督学习方法，用于使用大量未标记数据来训练模型。将围绕基础设施损坏预测模型的多模式数据融合这一广泛领域建立一个新的知识库。将回答有关统一表示，跨模态和数据融合之间的转换和对齐的具体问题。一种新型的灾害不可知的基础设施损坏预测模型也将从这项研究中出现。这样的模型将具有接收一种或多种类型的输入模态（即，图像、文本和工程/表格数据），并产生与因果事件的类型无关的基础设施损坏水平（例如，地震或飓风）。使用来自多个自然灾害事件的综合数据集（飓风和地震），该项目将包括实验，以揭示新的光的能力，具体的灾害和灾害不可知的多式联运模式，以提高早期-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准。

Immediately after a major hazard event (e.g., wildfire, earthquake, flood), a prompt assessment of the geographic distribution and severity of infrastructure damage is vital to the success of the emergency response and early recovery planning. This situational awareness is an important part of the decision-making processes that are implemented by facility owners, users, emergency responders and local and state officials. Conversely, a general lack of knowledge about the impacted state of the built environment can lead to a disorganized public response and slower recovery. While a comprehensive assessment of the extent and distribution of infrastructure damage can be obtained from in-person inspections conducted by building professionals, depending on the scale of the event, this can be a lengthy, resource intensive process. This Disaster Resilience Research Grants (DRRG) project will address this challenge by utilizing principles from artificial intelligence (AI) to develop near real-time infrastructure damage prediction models that can process and utilize different types of data and information (e.g., images, text, tabular data). By advancing our ability to effectively integrate disparate information sources, this project aims to transform the way that physical damage to infrastructure is estimated in the aftermath of a major disaster event, thereby enhancing the emergency response and recovery planning phases that follow. The research will provide training for doctoral and masters students and an opportunity to teach undergraduates from different backgrounds how science and engineering coupled with AI technologies can be used to improve community response to extreme events.Fundamental concepts and methodological advancements in multimodal learning will be used to transform and enhance infrastructure damage prediction models for use in the immediate post-event environment. The state-of-the-art in image-based damage assessment will be advanced along two dimensions: (1) developing Vision Transformer-based methods and (2) establishing a self-supervised learning methodology for training the models using large collections of unlabeled data. A new knowledge base will be established around the broad area of multimodal data fusion for infrastructure damage prediction models. Specific questions regarding unified representation, translation across and alignment between modalities and data fusion will be answered. A new type of hazard-agnostic infrastructure damage prediction model will also emerge from this research. Such a model will have the ability to receive an integrated representation of one or more types of input modalities (i.e., image, text, and engineering/tabular data) and produce, as output, an infrastructure damage level that is agnostic to the type of causal event (e.g., earthquake or hurricane). Using a comprehensive data set from multiple natural hazard events (hurricane and earthquake), the project will include experiments to shed new light on the ability of both hazard-specific and hazard-agnostic multimodal models to enhance early-stage infrastructure damage assessments for increased situational awareness and enhanced resilience.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.