Collaborative Research: Elements: VLCC-States: Versioned Lineage-Driven Checkpointing of Composable States

检查点是各种科学应用程序在小型和大型计算规模上使用的基本模式。被长期运行的应用程序广泛采用，以实现弹性目的（即，checkpoint-restart），它已经看到了额外的用例的爆炸式增长，这些用例直接帮助应用程序更快地前进，甚至在没有故障的情况下也能减少解决问题的时间：伴随计算（在金融建模，天气预测，计算流体动力学，地震成像和控制理论中必不可少）需要在向前传递中捕获检查点的历史，然后在向后传递中重新访问。训练人工智能模型，越来越多地被科学应用所使用，通常会导致不会导致收敛或可能导致不期望的模式的轨迹，从而促使需要回溯到学习模型的早期检查点以尝试替代方案。迁移学习和使用学习模型的前一个检查点进行微调可以用来更快地适应训练，避免从头开始进行昂贵的训练。许多其他用例在科学计算中也很重要：挂起-恢复（例如，抢占长时间运行的作业以支持更高优先级的作业）、迁移（在一台机器上检查点，在另一台机器上重新启动）、调试（重放有问题的代码区域以再现错误，而无需从头开始）和再现性（在重复运行期间检查点并比较中间数据）。尽管具有广泛的适用性，但当前最先进的解决方案缺乏有效解决这些场景所需的灵活性、性能和可扩展性。Versioned Lineage-Driven Checkpointing of Composable States（VLCC-States）项目旨在填补这一空白。它将简化用于科学应用的检查点模式的开发和使用，从而简化和提高跨不同社区的集成工作的可重用性，提高对大量检查点场景的认识，减少开发工作和成本，并实现灵活的定制，以针对所需的应用场景提取最佳性能和可扩展性。VLCC-States在三个领域提供技术创新。首先，它引入了中间状态的可组合提供程序，隐藏了捕获和组装分布式数据结构的检查点及其跨不同模块和编程语言的转换的复杂性，同时优化了它们的布局，以消除冗余，减小大小并提高性能。其次，它提供了多级协同优化的缓存和预取技术，这使得可扩展的管理的生命周期的检查点的交叉捕获和重用操作的异构存储堆栈下并发。第三，它为大型人工智能模型开发专门的检查点工具，重点是与PyTorch和DeepSpeed的集成，使用户能够使用熟悉的API透明地利用高性能和可扩展的检查点。该项目将邀请行业和国家研究实验室的合作伙伴共同设计VLCC国家，调整其能力并评估其实施情况。该项目将开展教育和扩大参与活动，以提高社区对科学数据管理挑战的认识和理解。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Checkpointing is a fundamental pattern used by a variety of scientific applications at both small and large computing scales. Widely adopted for resilience purposes by long-running applications (i.e., checkpoint-restart), it has seen an explosion of additional use cases that directly help applications progress faster and reduce time-to-solution even in the absence of failures: adjoint computations (essential in financial modeling, weather prediction, computational fluid dynamics, seismic imaging, and control theory) need to capture a history of checkpoints in a forward pass, which are then revisited in a backward pass. Training artificial intelligence models, increasingly used by scientific applications, often results in trajectories that do not lead to convergence or may lead to undesirable patterns, prompting the need to backtrack to an earlier checkpoint of the learning model to try an alternative. Transfer learning and fine-tuning using a previous checkpoint of a learning model can be used to adapt the training more quickly, avoiding expensive training from scratch. Many other use cases are important in scientific computing: suspend-resume (e.g., to preempt a long-running job in favor of a higher priority job), migration (checkpoint on one machine, restart on another), debugging (replay a problematic code region to reproduce errors without starting from scratch), and reproducibility (checkpoint and compare intermediate data during repeated runs). Despite broad applicability, current state-of-the-art solutions lack the flexibility, performance, and scalability needed to address these scenarios efficiently. The Versioned Lineage-Driven Checkpointing of Composable States (VLCC-States) project aims to fill this gap. It will streamline the development and use of checkpointing patterns for scientific applications, which simplifies and improves the reusability of integration efforts across different communities, improves awareness of the multitude of checkpointing scenarios, reduces development effort and cost, and enables flexible customization to extract the best performance and scalability for the desired application scenario.VLCC-States provides technical innovation in three areas. First, it introduces composable providers of intermediate states, which hide the complexity of capturing and assembling checkpoints of distributed data structures and their transformations across different modules and programming languages while optimizing their layout to eliminate redundancies, reduce sizes, and improve performance. Second, it provides multi-level co-optimized caching and prefetching techniques, which enable scalable management of the life cycle of checkpoints for interleavings of capture and reuse operations on heterogeneous storage stacks under concurrency. Third, it develops specialized checkpointing tools for large Artificial Intelligence models, with a focus on integration with PyTorch and DeepSpeed, to enable users to transparently take advantage of high-performance and scalable checkpointing using a familiar API. This project will engage partners in industry and national research laboratories to co-design VLCC-States, tune its capabilities, and evaluate its implementation. This project will undertake educational and broadening participation activities to improve community awareness and understanding of challenges in scientific data management.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.