CSR: Small: Accelerating Data Intensive Scientific Workflows with Consistency Contracts

科学计算中的高级发现越来越依赖于复杂工作流的成功执行，这些工作流将多个应用程序组合在一起，以在大型高性能集群上协同运行。在这种情况下，一个普遍的挑战是共享并行文件系统的性能。 由于每个文件系统交互在性能和一致性方面都有非常不同的需求，因此文件系统必须遵循最保守的方法来处理最坏的情况。 因此，很少达到峰值性能。我们提出“一致性合同”作为解决这个问题的方案。 这种新颖的方法要求整个工作流在每次执行开始时声明文件系统的预期用途，从而允许运行时系统执行各种优化。该项目将通过构建一个实验系统（Pledge）来评估一致性合同的概念，该系统将在现有的数据密集型工作流程上启用和执行合同，并将对当前实践的干扰降至最低。我们假设，这种方法将显着提高高性能集群上运行的数据密集型科学应用程序的性能，并有可能得到更广泛的应用。我们对一致性的关注来自于我们的观察，即今天的共享并行文件系统被要求扮演多个角色：移动大文件，交付复杂的软件树，在任务之间提供缓冲区，以及在任务之间提供同步。当前的文件系统提供了最保守的顺序一致性来处理最坏的情况。 而不是依赖于共享的文件系统来执行最后一分钟的运行时仲裁的每个单独的文件系统操作，我们认为，作为一个整体的工作流应该声明其意图的执行期间，指示的路径，访问模式，和一致性要求所需的整个工作流运行。 有了合同在手，运行时系统可以执行各种优化，利用集群的内部存储和I/O容量作为一个整体，例如利用精简的方法进行只读访问。我们假设，工作流级的一致性管理将产生更高的有效I/O带宽和事务速率比严格的全局一致性管理的数据密集型科学应用程序运行在高性能集群。这些改进的I/O速率将转化为更快的端到端运行时间和更少的最终用户和系统管理员的意外性能故障。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Advanced discovery in scientific computing increasingly depends upon the successful execution of complex workflows that combine multiple applications together to run in concert on a large high performance cluster. A widespread challenge in this setting is the performance of the shared parallel filesystem. Because each file system interaction has very different needs in terms of performance and consistency, the filesystem is obliged to follow the most conservative approach to handle the worst case. As a result, peak performance is rarely achieved. We propose "consistency contracts" as the solution to this problem. This novel approach requires the workflow as a whole to declare its intended uses of the file system at the start of each execution, allowing the runtime system to perform a variety of optimizations. This project will evaluate the concept of consistency contracts by constructing an experimental system (Pledge) that enables and enforces contracts on existing data intensive workflows, with minimal disruption to current practice. We hypothesize that this approach will significantly improve performance for data intensive scientific applications running on high performance clusters,and has the potential to be more widely applied. Our focus on consistency results from our observation that today's shared parallel filesystem is asked to fill multiple roles: moving large files, delivering complex software trees, providing buffers between tasks, and providing synchronization between tasks. Current filesystems provide the most conservative sequential consistency to handle the worst case. Rather than depend upon the shared filesystem to perform last-minute runtime arbitration of every individual filesystem operation, we argue that the workflow as a whole should declare its intentions for the duration of the execution, indicating the paths, access modes, and consistency requirements needed for the entire workflow run. With a contract in hand, the runtime system can then perform a variety of optimizations that exploit the internal storage and I/O capacity of the cluster as a whole, for example utilizing a streamlined approach for read-only access. We hypothesize that workflow-level consistency management will yield higher effective I/O bandwidth and transaction rates than strict global consistency management for data intensive scientific applications running on high performance clusters. These improved I/O rates will translate into faster end-to-end runtimes and fewer unexpected performance failures for end users and system administrators.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.