CSR: Small: Accelerating Data Intensive Scientific Workflows with Consistency Contracts

科学计算中的高级发现越来越依赖于复杂工作流的成功执行，这些工作流将多个应用程序组合在一起，在大型高性能集群上协同运行。在这种设置中，一个普遍的挑战是共享并行文件系统的性能。因为每个文件系统交互在性能和一致性方面有非常不同的需求，所以文件系统必须遵循最保守的方法来处理最坏的情况。因此，很难达到最佳性能。我们提出“一致性合约”作为这个问题的解决方案。这种新颖的方法要求整个工作流在每次执行开始时声明文件系统的预期用途，从而允许运行时系统执行各种优化。该项目将通过构建一个实验系统（Pledge）来评估一致性合同的概念，该系统可以在现有的数据密集型工作流程上启用和执行合同，同时对当前实践的干扰最小。我们假设这种方法将显著提高运行在高性能集群上的数据密集型科学应用程序的性能，并且具有更广泛应用的潜力。我们对一致性的关注源于我们的观察，即今天的共享并行文件系统被要求扮演多种角色：移动大文件、交付复杂的软件树、在任务之间提供缓冲区以及在任务之间提供同步。当前的文件系统提供最保守的顺序一致性来处理最坏的情况。与其依赖于共享文件系统对每个单独的文件系统操作执行最后一分钟的运行时仲裁，我们认为工作流作为一个整体应该声明它在执行期间的意图，指示整个工作流运行所需的路径、访问模式和一致性需求。有了契约，运行时系统就可以执行各种优化，利用整个集群的内部存储和I/O容量，例如利用用于只读访问的流线型方法。我们假设，对于运行在高性能集群上的数据密集型科学应用程序，工作流级一致性管理将比严格的全局一致性管理产生更高的有效I/O带宽和事务率。这些改进的I/O速率将为最终用户和系统管理员带来更快的端到端运行时间和更少的意外性能故障。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Advanced discovery in scientific computing increasingly depends upon the successful execution of complex workflows that combine multiple applications together to run in concert on a large high performance cluster. A widespread challenge in this setting is the performance of the shared parallel filesystem. Because each file system interaction has very different needs in terms of performance and consistency, the filesystem is obliged to follow the most conservative approach to handle the worst case. As a result, peak performance is rarely achieved. We propose "consistency contracts" as the solution to this problem. This novel approach requires the workflow as a whole to declare its intended uses of the file system at the start of each execution, allowing the runtime system to perform a variety of optimizations. This project will evaluate the concept of consistency contracts by constructing an experimental system (Pledge) that enables and enforces contracts on existing data intensive workflows, with minimal disruption to current practice. We hypothesize that this approach will significantly improve performance for data intensive scientific applications running on high performance clusters,and has the potential to be more widely applied. Our focus on consistency results from our observation that today's shared parallel filesystem is asked to fill multiple roles: moving large files, delivering complex software trees, providing buffers between tasks, and providing synchronization between tasks. Current filesystems provide the most conservative sequential consistency to handle the worst case. Rather than depend upon the shared filesystem to perform last-minute runtime arbitration of every individual filesystem operation, we argue that the workflow as a whole should declare its intentions for the duration of the execution, indicating the paths, access modes, and consistency requirements needed for the entire workflow run. With a contract in hand, the runtime system can then perform a variety of optimizations that exploit the internal storage and I/O capacity of the cluster as a whole, for example utilizing a streamlined approach for read-only access. We hypothesize that workflow-level consistency management will yield higher effective I/O bandwidth and transaction rates than strict global consistency management for data intensive scientific applications running on high performance clusters. These improved I/O rates will translate into faster end-to-end runtimes and fewer unexpected performance failures for end users and system administrators.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.