HCL: Distributing Parallel Data Structures in Extreme Scales
HCL: Distributing Parallel Data Structures in Extreme Scales
复制标题
DOI：
10.1109/cluster49012.2020.00035
复制
发表时间：
2020-09
期刊：
影响因子：
--
通讯作者：
H. Devarajan;Anthony Kougkas;Keith Bateman;Xian-He Sun
中科院分区：
文献类型：
--
作者：
H. Devarajan;Anthony Kougkas;Keith Bateman;Xian-He Sun
大多数并行程序使用不规则的控制流和数据结构，这非常适合单方面的通信范例，例如 MPI 或 PGAS 编程语言。然而，这些环境缺乏高效的基于功能的应用程序库，这些应用程序库可以利用流行的通信结构，例如 TCP、Infinity Band (IB) 和融合以太网 RDMA (RoCE)。此外，缺乏高性能的数据结构接口。我们推出 Hermes 容器库 (HCL)，这是一个高性能分布式数据结构库，提供高级抽象，包括哈希映射、集合和队列。 HCL 使用基于 RDMA 的 RPC 技术来实现新颖的过程编程范例。在本文中，我们认为基于 RDMA 的 RPC 技术可以作为高性能、灵活且无需协调的后端来实现复杂的数据结构。测试实际工作负载的评估结果表明，与最先进的分布式数据结构库 BCL 相比，HCL 程序的速度快 2 倍到 12 倍。
Most parallel programs use irregular control flow and data structures, which are perfect for one-sided communication paradigms such as MPI or PGAS programming languages. However, these environments lack the presence of efficient function-based application libraries that can utilize popular communication fabrics such as TCP, Infinity Band (IB), and RDMA over Converged Ethernet (RoCE). Additionally, there is a lack of high-performance data structure interfaces. We present Hermes Container Library (HCL), a high-performance distributed data structures library that offers high-level abstractions including hash-maps, sets, and queues. HCL uses a RPC over RDMA technology that implements a novel procedural programming paradigm. In this paper, we argue a RPC over RDMA technology can serve as a high-performance, flexible, and co-ordination free backend for implementing complex data structures. Evaluation results from testing real workloads shows that HCL programs are 2x to 12x faster compared to BCL, a state-of-the-art distributed data structure library.