Decaf: Decoupled Dataflows for In Situ High-Performance Workflows
Decaf: Decoupled Dataflows for In Situ High-Performance Workflows
复制标题
Decaf：用于现场高性能工作流程的解耦数据流
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
T. Peterka
中科院分区：
文献类型：
--
作者：
Matthieu Dreher;T. Peterka
Decaf 是一个数据流系统，用于 HPC 工作流程中耦合任务的并行通信。数据流可以执行任意数据转换，范围从简单的数据转发到复杂的数据重新分配。 Decaf 通过允许用户分配资源并在数据流中执行自定义代码来实现此目的。通过数据流的所有通信都是通过 MPI 传递的高效并行消息。调用任务的运行时完全是消息驱动的； Decaf 在收到任务的所有消息后执行该任务。这种消息驱动的运行时允许工作流图中的循环任务依赖性，例如，基于下游任务的结果来制定计算指导。 Decaf 包含一个简单的 Python API，用于描述工作流程图。这使得 Decaf 能够独立作为一个完整的工作流系统，但 Decaf 也可以被一个或多个其他工作流系统用作数据流层，形成基于任务的异构计算环境。在一项实验中，我们使用 FlowVR 和 Damaris 工作流程系统以及 Decaf 作为数据流，将分子动力学代码与可视化工具结合起来。在另一个实验中，我们测试了宇宙学代码与 Voronoi 曲面细分和密度估计代码的耦合，使用 MPI 进行模拟，使用 DIY 编程模型进行两个分析代码，并使用 Decaf 进行数据流。这种由异构软件基础设施组成的工作流程的存在是因为组件是使用不同的编程模型和运行时单独开发的，这是第一次在 HPC 系统上现场演示不同组件的异构耦合。
Decaf is a dataflow system for the parallel communication of coupled tasks in an HPC workflow. The dataflow can perform arbitrary data transformations ranging from simply forwarding data to complex data redistribution. Decaf does this by allowing the user to allocate resources and execute custom code in the dataflow. All communication through the dataflow is efficient parallel message passing over MPI. The runtime for calling tasks is entirely message-driven; Decaf executes a task when all messages for the task have been received. Such a messagedriven runtime allows cyclic task dependencies in the workflow graph, for example, to enact computational steering based on the result of downstream tasks. Decaf includes a simple Python API for describing the workflow graph. This allows Decaf to stand alone as a complete workflow system, but Decaf can also be used as the dataflow layer by one or more other workflow systems to form a heterogeneous task-based computing environment. In one experiment, we couple a molecular dynamics code with a visualization tool using the FlowVR and Damaris workflow systems and Decaf for the dataflow. In another experiment, we test the coupling of a cosmology code with Voronoi tessellation and density estimation codes using MPI for the simulation, the DIY programming model for the two analysis codes, and Decaf for the dataflow. Such workflows consisting of heterogeneous software infrastructures exist because components are developed separately with different programming models and runtimes, and this is the first time that such heterogeneous coupling of diverse components was demonstrated in situ on HPC systems.