Elements: Streaming Molecular Dynamics Simulation Trajectories for Direct Analysis: Applications to Sub-Picosecond Dynamics in Microsecond Simulations

计算能力的大规模增长正在增加数值模拟中生成的数据量，而数据存储和传输能力却没有以同样的速度增长。这对于全原子分子动力学（MD）模拟尤其如此，该模拟用于在包含数十万至数百万个原子的系统中研究生物分子和新材料在微秒时间尺度上的功能。这种模拟评估了109个不同时间的原子位置和速度，这相当于2 PB的数据，超过了合理的数据存储和传输能力。因此，模拟数据仅以粗略的时间间隔存储，即使这种方法丢失了对计算许多实验观测值至关重要的快速分子过程的信息。该项目中创建的软件基础设施通过数据生成和分析的融合避免了分子动力学模拟中的此类信息丢失。它不是存储数据进行后处理（现状），而是通过流协议直接传递到并行软件平台。将流协议实施到具有大用户群的现有软件包中允许直接集成到已建立的模拟协议中。因此，分子动力学模拟产生更多的可用信息，并更有效地利用相关的计算资源。MD模拟的流协议使用TCP/IP套接字应用程序编程接口（API）将数据直接从运行的模拟传输到分析软件。这种方法可以在分子模拟中同时分析快速（亚皮秒）和慢速（微秒）过程，而不会产生瓶颈或需要大量的轨迹输出文件。分析是用开源MDAnalysis平台执行的，该平台的工具被实现为MDAkit插件，这些插件从流接口中受益匪浅，例如3D-2 PT（空间分辨两相热力学）和相关工具的空间分辨蛋白质溶剂化图的分析。这个奖项由NSF高级网络基础设施办公室联合支持的化学部。这个奖项反映了NSF的法定基金会的使命是履行其使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评价，被认为值得支持。

Massive increases in computing capabilities are increasing the data volume generated in numerical simulations, while data storage and transfer capabilities are not growing at the same pace. This is especially true for all-atom molecular dynamics (MD) simulations that are used to study the function of biomolecules and novel materials on timescales of microseconds in systems containing hundreds of thousands to millions of atoms. Such simulations evaluate atom positions and velocities for 109 distinct times, which corresponds to 2 petabytes of data and exceeds reasonable data storage and transfer capacities. As a result, simulation data is stored only at coarse time intervals even though this approach loses information on fast molecular processes that is essential to computing many experimental observables. The software infrastructure created in this project avoids such information loss in molecular dynamics simulations via convergence of data generation and analysis. Instead of storing data for post-processing (status quo), it is passed directly to a parallel software platform via a streaming protocol. The implementation of the streaming protocol into existing software packages with a large user base allows for a direct integration into established simulation protocols. As a result, molecular dynamics simulations produce more usable information, and associated computational resources are used more efficiently.The streaming protocol for MD simulations uses a TCP/IP socket application programming interface (API) to transfer data directly from a running simulation to the analysis software. This approach enables the simultaneous analysis of fast (sub-picosecond) and slow (microsecond) processes in molecular simulations without creating bottlenecks or requiring massive trajectory output files. The analysis is performed with the open-source MDAnalysis platform for which tools are implemented as MDAKit plugins that significantly benefit from the streaming interface, such as 3D-2PT (spatially resolved two-phase thermodynamics) and related tools for the analysis of spatially resolved protein solvation maps.This award by the NSF Office of Advanced Cyberinfrastructure is jointly supported by the Division of Chemistry.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.