Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler Augmentation
Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler Augmentation
复制标题
DOI：
10.1109/sc41404.2022.00065
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
William S. Moses;S. Narayanan;Ludger Paehler;Valentin Churavy;Michel Schanen;J. Hückelheim;J. Doerfert;P. Hovland
中科院分区：
文献类型：
--
作者：
William S. Moses;S. Narayanan;Ludger Paehler;Valentin Churavy;Michel Schanen;J. Hückelheim;J. Doerfert;P. Hovland
导数对于众多科学、工程和机器学习应用至关重要。虽然现有工具能为单一语言的程序生成导数，但现代并行应用结合了一系列框架和语言，以便在不断发展的硬件环境中利用可用的性能和功能。我们提出了一种区分任意基于有向无环图（DAG）的并行性的方案，该方案保持了可扩展性和效率，并在基于LLVM的Enzyme自动微分框架中得以实现。通过与一个成熟的编译器后端集成，Enzyme能够区分众多并行框架并直接控制代码生成。结合其区分任何基于LLVM的语言的能力，这种灵活性使Enzyme能够利用编译器工具链进行并行和特定于微分的优化。我们对用不同编程语言（C++、Julia）和并行框架（OpenMP、MPI、RAJA、Julia任务、MPI.jl）编写的LULESH和miniBUDE应用的九个不同版本进行微分，展示出与原始程序相似的可扩展性。在具有64个线程或节点的基准测试中，我们发现C++的微分开销为3.4 - 6.8倍，Julia为5.4 - 12.5倍。
Derivatives are key to numerous science, engineering, and machine learning applications. While existing tools generate derivatives of programs in a single language, modern parallel applications combine a set of frameworks and languages to leverage available performance and function in an evolving hardware landscape. We propose a scheme for differentiating arbitrary DAG-based parallelism that preserves scalability and efficiency, implemented into the LLVM-based Enzyme automatic differentiation framework. By integrating with a full-fledged compiler backend, Enzyme can differentiate numerous parallel frameworks and directly control code generation. Combined with its ability to differentiate any LLVM-based language, this flexibility permits Enzyme to leverage the compiler tool chain for parallel and differentiation-specitic optimizations. We differentiate nine distinct versions of the LULESH and miniBUDE applications, written in different programming languages (C++, Julia) and parallel frameworks (OpenMP, MPI, RAJA, Julia tasks, MPI.jl), demonstrating similar scalability to the original program. On benchmarks with 64 threads or nodes, we find a differentiation overhead of 3.4–6.8× on C++ and 5.4–12.5× on Julia.