CDS&E: In Situ Data Analysis and Scalable Machine Learning for Exascale Scientific Simulations

大规模科学模拟应用于材料科学、气候建模、燃烧等领域。这些模拟通常受到运行它们的硬件的限制，包括计算平台和存储系统的容量。这些模拟的目标可能包括在模拟输出中发现罕见但有趣的事件，发现常见的事件序列，以及发现事件之间的因果关系。通常，这些科学模拟在模拟运行期间会消耗高性能计算平台上所有可用的计算资源，并且被迫只使用采样数据技术来减小模拟的大小，以便能够存储、传输和后处理输出数据。这种数据抽样降低了科学结果的质量，因为在分析过程中并不是所有可用的数据都被利用。该项目旨在利用创新的“原位”算法和罕见事件检测的机器学习技术，大大提高科学模拟结果的规模和质量。这项研究将通过大规模的材料科学模拟来验证，即能够在恶劣的化学环境和高温/高压操作条件下感知和修复损伤的自修复纳米材料系统。自我修复的重要性在于，它可以提高材料的可靠性和使用寿命，同时降低高温涡轮机、风能、太阳能和照明系统的制造、监测和维护成本。该研究可以推广到一系列科学模拟领域，这些领域的共同目标是发现罕见和有趣的事件、事件序列和事件之间的因果关系。最后，研究理念和成果将被纳入研究团队教授的研究生水平课程。该项目的目标是证明研究方法的可行性、性能和可扩展性，在一个定义良好、可重用的原位软件框架内，使用原位机器学习算法大大提高了百亿亿次科学模拟的质量。项目范围包括：选择一种简化但具有代表性的、适合超状态平行复制动力学（SPRD）的长时间材料工艺；超态跃迁罕见事件检测的原位机器学习算法研究并研究基于库的方法来支持百亿亿次模拟与原位机器学习算法的高性能耦合。为了实现项目目标，确定了以下三个目标：1)证明原位SPRD模拟用于预测长时间材料过程的可行性、性能和可扩展性；2)证明了超状态转换稀有事件检测的原位机器学习算法的可行性、性能和可扩展性；3)证明基于原位库的方法耦合百亿亿次模拟和机器学习算法的可行性、性能和可扩展性。

Large scale scientific simulations are used in a range of application domains including materials science, climate modeling, combustion and others. These simulations are often limited by the hardware on which they run, including the capacity of computational platforms and storage systems. The goals of these simulations may include finding rare but interesting events in simulation output, discovering common sequences of events, and discovering causality among events. Often, these scientific simulations consume all available computational resources on a high performance computing platform during a simulation run, and be forced to only sample data techniques to decrease the size of the simulation so as to make it possible to store, transfer and post-process the output data. Such data sampling reduces the quality of science results, since not all available data are utilized during analysis. This project aims to greatly improve the scale and quality of scientific simulation results using innovative "in situ" algorithms and machine learning techniques for rare event detection. This research will be validated using a large-scale materials science simulation, that of self-healing nanomaterial system capable of sensing and repairing damage in harsh chemical environments and in high temperature/high pressure operating conditions. Self-healing is of significance since it can improve the reliability and lifetime of materials while reducing the cost of manufacturing, monitoring and maintenance of high-temperature turbines, wind, solar energy and lighting systems. The research can be generalized to a range of scientific simulation domains that share the common goals of discovering rare and interesting events, sequences of events and causality among events. Finally, the research concepts and results will be incorporated into graduate level courses taught by the research team.The goal of the project is to demonstrate the feasibility, performance and scalability of the research approaches in greatly improving the quality of exascale scientific simulations using in situ machine learning algorithms within a well-defined, reusable in situ software framework. The scope of the project includes: selecting a simplified, but representative, long-time material process suitable for super-state parallel replica dynamics (SPRD); developing in situ machine learning algorithms for rare event detection of super-state transitions; and studying library-based approaches to support the high performance coupling of exascale simulations with in situ machine learning algorithms. To accomplish the project goals, the following three objectives are defined: 1) Prove the feasibility, performance and scalability of in situ SPRD simulation for predicting long-time material processes; 2) Prove the feasibility, performance and scalability of in situ machine learning algorithms for rare-event detection of super-state transitions; and 3) Prove the feasibility, performance and scalability of in situ library-based approaches to coupling exascale simulations and machine learning algorithms.