Multimodal Disaster Impact Assessment Models for Enhanced Resilience

在重大灾害事件（如野火、地震、洪水）发生后，立即对基础设施受损的地理分布和严重程度进行评估，对于成功开展应急反应和早期恢复规划至关重要。这种态势感知是设施所有者、用户、应急响应人员以及地方和州官员实施决策过程的重要组成部分。相反，对建筑环境受影响状态的普遍缺乏了解可能导致公众反应混乱和恢复缓慢。虽然对基础设施损坏程度和分布的全面评估可以由建筑专业人员亲自进行，但根据事件的规模，这可能是一个漫长的、资源密集的过程。该灾害恢复研究资助（DRRG）项目将利用人工智能（AI）的原理开发近乎实时的基础设施损坏预测模型，该模型可以处理和利用不同类型的数据和信息（如图像、文本、表格数据），从而应对这一挑战。通过提高我们有效整合不同信息来源的能力，该项目旨在改变在重大灾害事件发生后估计基础设施物理损害的方式，从而加强随后的应急反应和恢复规划阶段。该研究将为博士生和硕士生提供培训，并有机会向来自不同背景的本科生教授如何利用科学和工程与人工智能技术相结合来改善社区对极端事件的反应。多模态学习的基本概念和方法进步将用于转换和增强基础设施损坏预测模型，以便在事件发生后立即使用。基于图像的最先进的损伤评估将沿着两个维度发展：(1)开发基于视觉转换器的方法；(2)建立一种自我监督的学习方法，用于使用大量未标记数据集训练模型。围绕多模态数据融合的广阔领域，为基础设施损伤预测模型建立新的知识库。将回答关于统一表示、模式之间的转换和对齐以及数据融合的具体问题。该研究还将产生一种新型的与灾害无关的基础设施损害预测模型。这样的模型将能够接收一种或多种输入模式（即图像、文本和工程/表格数据）的集成表示，并产生与因果事件类型（例如地震或飓风）无关的基础设施损坏水平作为输出。利用来自多种自然灾害事件（飓风和地震）的综合数据集，该项目将包括实验，以揭示特定灾害和不可知灾害的多模式模型的新能力，以加强早期基础设施损害评估，从而提高态势感知和增强复原力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Immediately after a major hazard event (e.g., wildfire, earthquake, flood), a prompt assessment of the geographic distribution and severity of infrastructure damage is vital to the success of the emergency response and early recovery planning. This situational awareness is an important part of the decision-making processes that are implemented by facility owners, users, emergency responders and local and state officials. Conversely, a general lack of knowledge about the impacted state of the built environment can lead to a disorganized public response and slower recovery. While a comprehensive assessment of the extent and distribution of infrastructure damage can be obtained from in-person inspections conducted by building professionals, depending on the scale of the event, this can be a lengthy, resource intensive process. This Disaster Resilience Research Grants (DRRG) project will address this challenge by utilizing principles from artificial intelligence (AI) to develop near real-time infrastructure damage prediction models that can process and utilize different types of data and information (e.g., images, text, tabular data). By advancing our ability to effectively integrate disparate information sources, this project aims to transform the way that physical damage to infrastructure is estimated in the aftermath of a major disaster event, thereby enhancing the emergency response and recovery planning phases that follow. The research will provide training for doctoral and masters students and an opportunity to teach undergraduates from different backgrounds how science and engineering coupled with AI technologies can be used to improve community response to extreme events.Fundamental concepts and methodological advancements in multimodal learning will be used to transform and enhance infrastructure damage prediction models for use in the immediate post-event environment. The state-of-the-art in image-based damage assessment will be advanced along two dimensions: (1) developing Vision Transformer-based methods and (2) establishing a self-supervised learning methodology for training the models using large collections of unlabeled data. A new knowledge base will be established around the broad area of multimodal data fusion for infrastructure damage prediction models. Specific questions regarding unified representation, translation across and alignment between modalities and data fusion will be answered. A new type of hazard-agnostic infrastructure damage prediction model will also emerge from this research. Such a model will have the ability to receive an integrated representation of one or more types of input modalities (i.e., image, text, and engineering/tabular data) and produce, as output, an infrastructure damage level that is agnostic to the type of causal event (e.g., earthquake or hurricane). Using a comprehensive data set from multiple natural hazard events (hurricane and earthquake), the project will include experiments to shed new light on the ability of both hazard-specific and hazard-agnostic multimodal models to enhance early-stage infrastructure damage assessments for increased situational awareness and enhanced resilience.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.