NSCI: SI2-SSE: An Extensible Model to Support Scalable Checkpoint-Restart for DMTCP Across Multiple Disciplines

检查点是一种定期将长时间运行的计算机程序的状态保存到磁盘的技术。如果在程序运行期间发生计算机崩溃，则可以从先前保存在磁盘上的“检查点”文件中重新启动程序状态。这个项目的目标是发现、实现和部署适应检查点的新技术，从而提供更强大的能力，可以在支持各种科学和工程学科研究的应用程序之间轻松使用。特别是，经典（透明）检查点模型的一个问题是，这些包没有建模，因此不能在重新启动时重新创建原始程序与其他外部进程或程序之间的通信。在这个项目中，将为常用的通信机制开发一个虚拟化模型，以便在重启时模拟外部通信。检查点在学术界、工业界和政府中广泛使用，特别是那些长期运行的高性能计算程序。因此，项目成果具有广泛的适用性和价值。该项目还有一个额外的好处，那就是教育下一代学生掌握宝贵的、高度可转移的系统技能。目前，透明检查点重新启动主要用于容错，并且主要用于没有外部通信的封闭系统。DMTCP是一个有12年历史的开源检查点项目。它目前正在发展的检查点流程虚拟化模型使应用程序能够支持与外部子系统交互的复杂应用程序。该项目探索并扩展了流程虚拟化模型，以便使检查点重新启动适应多个新颖的应用程序，并扩展其在多个科学和工程学科中的使用。受益的学科包括：超级计算（特别是，为实际的百亿亿次检查点铺平道路）；适应当前工作负载的计算机集群灵活资源管理器（批处理队列）的新策略更好地支持电子设计自动化（EDA）的硬件电路仿真器。示例挑战包括需要在较新的低延迟网络（如Omni-Path）上支持透明检查点，将特定于应用程序的检查点与透明dmtcp风格的检查点集成，需要在高端集群的检查点期间避免“泛滥”后端存储，以及受益于通过检查点任意暂停运行作业的灵活性的新型资源管理器。这项工作将提供一个简单的模型，允许最终用户轻松地构建他们自己的扩展，以支持外部子系统的检查点，而不是为上面的每一个构建特别的解决方案。这个简单的模型将通过对上面描述的许多示例挑战的解决方案进行泛化而得到。除了容错性之外，该技术还具有以下优点：快速启动（进程初始化后的检查点，以便在以后的会话中重新启动并跳过此阶段）；调试（例如每30秒检查点一次）；可重现的bug报告；延长互动环节（如晚餐前的检查点和第二天的重新开始）；等等......。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Checkpointing is a technique that periodically saves the state of a long-running computer program to disk. If a computer crash occurs during the running of the program, one can then restart the program state from a previously saved "checkpoint" file on disk. The goal of this project is to discover, implement and deploy novel techniques for adapting checkpointing so as to provide a more robust capability easily usable across applications supporting the research of a variety of scientific and engineering disciplines. In particular, a problem with the classic (transparent) checkpoint model is that these packages do not model, and hence cannot recreate upon restart, communications between the original program and other external processes or programs. In this project, a virtualization model for commonly used mechanisms for communication will be developed so that on restart, external communications are emulated. Checkpointing is used across academia, industry, and government, particularly by those with long-running high performance computing programs. Thus, the project outcomes have broad applicability and value. The project has the added benefit of educating the next generation of students in valuable and highly transferable system skills. Today, transparent checkpoint-restart today is used primarily for fault tolerance, and primarily in closed systems with no external communication. DMTCP is a twelve-year old open source checkpointing project. Its currently evolving process virtualization model of checkpointing enables an application to support complex applications that interact with external subsystems. The project explores and extends a model of process virtualization in order to adapt checkpoint-restart to multiple, novel applications, and to extend its use across multiple scientific and engineering disciplines. Example disciplines that will benefit include: supercomputing (and in particular, forging a path toward practical exascale checkpointing); novel strategies for flexible resource managers (batch queues) for computer clusters that adapt to the current workload; and better support for hardware circuit emulators for Electronic Design Automation (EDA). Example challenges include the need to support transparent checkpointing over the newer low-latency networks such as Omni-Path, integration of application-specific checkpointing with transparent DMTCP-style checkpointing, the need to avoid "flooding" back-end storage during checkpointing in high-end clusters, and new types of resource managers that benefit from the flexibility of arbitrarily suspending running jobs through checkpointing. Rather than build ad hoc solutions for each of the above, this work will provide a simple model allowing end users to easily build their own extensions to support checkpointing of the external subsystems. The simple model will be derived by generalizing over solutions to many of the example challenges described above. In addition to fault tolerance, the technology holds advantages for: fast startup (checkpoint after process initialization,in order to restart and skip this phase in future sessions); debugging (e.g. checkpoint every 30 seconds); reproducible bug reports; extended interactive sessions (e.g. checkpoint before dinner and restart the next day); and so on.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.