CSR: Small: Accelerating Data Intensive Scientific Workflows with Consistency Contracts

CSR：小：通过一致性合同加速数据密集型科学工作流程

• 批准号: 2317556
• 负责人: Douglas Thain
• 金额: $ 59.97万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2317556&HistoricalAwards=false
• 关键词: CSR; Small; Accelerating; Data; Intensive

Advanced discovery in scientific computing increasingly depends upon the successful execution of complex workflows that combine multiple applications together to run in concert on a large high performance cluster. A widespread challenge in this setting is the performance of the shared parallel filesystem. Because each file system interaction has very different needs in terms of performance and consistency, the filesystem is obliged to follow the most conservative approach to handle the worst case. As a result, peak performance is rarely achieved. We propose "consistency contracts" as the solution to this problem. This novel approach requires the workflow as a whole to declare its intended uses of the file system at the start of each execution, allowing the runtime system to perform a variety of optimizations. This project will evaluate the concept of consistency contracts by constructing an experimental system (Pledge) that enables and enforces contracts on existing data intensive workflows, with minimal disruption to current practice. We hypothesize that this approach will significantly improve performance for data intensive scientific applications running on high performance clusters,and has the potential to be more widely applied. Our focus on consistency results from our observation that today's shared parallel filesystem is asked to fill multiple roles: moving large files, delivering complex software trees, providing buffers between tasks, and providing synchronization between tasks. Current filesystems provide the most conservative sequential consistency to handle the worst case. Rather than depend upon the shared filesystem to perform last-minute runtime arbitration of every individual filesystem operation, we argue that the workflow as a whole should declare its intentions for the duration of the execution, indicating the paths, access modes, and consistency requirements needed for the entire workflow run. With a contract in hand, the runtime system can then perform a variety of optimizations that exploit the internal storage and I/O capacity of the cluster as a whole, for example utilizing a streamlined approach for read-only access. We hypothesize that workflow-level consistency management will yield higher effective I/O bandwidth and transaction rates than strict global consistency management for data intensive scientific applications running on high performance clusters. These improved I/O rates will translate into faster end-to-end runtimes and fewer unexpected performance failures for end users and system administrators.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学计算中的高级发现越来越依赖于复杂工作流的成功执行，这些工作流将多个应用程序组合在一起，以便在大型高性能集群上协调运行。这种设置中的一个普遍挑战是共享并行文件系统的性能。因为每个文件系统交互在性能和一致性方面都有非常不同的需求，所以文件系统必须遵循最保守的方法来处理最坏的情况。因此，很少能达到最佳性能。我们提出“一致性契约”作为这个问题的解决方案。这种新颖的方法要求工作流作为一个整体在每次执行开始时声明其对文件系统的预期用途，从而允许运行时系统执行各种优化。该项目将通过建立一个试验性系统(Pledge)来评估一致性合同的概念，该系统能够启用和执行现有数据密集型工作流程的合同，并最大限度地减少对当前做法的干扰。我们假设这种方法将显著提高在高性能集群上运行的数据密集型科学应用的性能，并具有更广泛应用的潜力。我们对一致性的关注源于我们的观察，即今天的共享并行文件系统被要求扮演多个角色：移动大型文件、交付复杂的软件树、在任务之间提供缓冲区以及在任务之间提供同步。当前的文件系统提供了最保守的顺序一致性来处理最坏的情况。与其依赖共享文件系统对每个单独的文件系统操作执行最后一刻的运行时仲裁，我们认为工作流作为一个整体应该声明其在执行持续时间内的意图，指示整个工作流运行所需的路径、访问模式和一致性要求。有了合同，运行时系统就可以执行各种优化，将集群的内部存储和I/O容量作为一个整体加以利用，例如利用只读访问的简化方法。我们假设，对于在高性能集群上运行的数据密集型科学应用程序，工作流级一致性管理将比严格的全局一致性管理产生更高的有效I/O带宽和事务率。这些改进的I/O速率将为最终用户和系统管理员带来更快的端到端运行时间和更少的意外性能故障。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。