Accelerating Apache Spark with FPGAs
Accelerating Apache Spark with FPGAs
复制标题
使用 FPGA 加速 Apache Spark
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
P. Chow
中科院分区：
文献类型：
--
作者：
Ehsan Ghasemi;P. Chow
Apache Spark已经成为最受欢迎的大数据处理引擎之一。Spark通过利用Java框架为大规模数据处理提供了一个平台无关的，高度抽象的编程范式。尽管Java提供了跨各种机器的软件可移植性，但它也限制了分布式环境（如Spark）的性能。虽然用更快的语言重写像Spark这样的平台可能是不现实的，但一种更可行的方法来减轻其糟糕的性能是加速计算，同时仍然在基于Java的框架内工作。本文演示了将现场可编程门阵列（FPGA）加速纳入Spark的可行性，并展示了我们的FPGA加速Spark环境的性能优势和瓶颈，该环境使用了k均值聚类算法的MapReduce实现，以表明即使使用性能没有很好优化的硬件平台，加速也是可能的。我们的方法的一个重要特征是，通过使用库函数，FPGA的使用对用户完全透明，这是用户访问Spark提供的函数的常见方式。高级用户可以使用高级综合进一步开发其他计算。
Apache Spark has become one of the most popular engines for big data processing. Spark provides a platform‐independent, high‐abstraction programming paradigm for large‐scale data processing by leveraging the Java framework. Though it provides software portability across various machines, Java also limits the performance of distributed environments, such as Spark. While it may be unrealistic to rewrite platforms like Spark in a faster language, a more viable approach to mitigate its poor performance is to accelerate the computations while still working within the Java‐based framework. This paper demonstrates the feasibility of incorporating Field‐Programmable Gate Array (FPGA) acceleration into Spark and presents the performance benefits and bottlenecks of our FPGA‐accelerated Spark environment using a MapReduce implementation of the k‐means clustering algorithm, to show that acceleration is possible even when using a hardware platform that is not well optimized for performance. An important feature of our approach is that the use of FPGAs is completely transparent to the user through the use of library functions, which is a common way by which users access functions provided by Spark. Power users can further develop other computations using high‐level synthesis.
影响因子：
4.6
作者：
J. Xu
通讯作者：
J. Xu