Accelerating Apache Spark Big Data Analysis with FPGAs
Accelerating Apache Spark Big Data Analysis with FPGAs
复制标题
使用 FPGA 加速 Apache Spark 大数据分析
DOI：
10.1109/uic-atc-scalcom-cbdcom-iop-smartworld.2016.0119
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
P. Chow
中科院分区：
文献类型：
--
作者：
Ehsan Ghasemi;P. Chow
Apache Spark已经成为最受欢迎的大数据处理引擎之一。Spark通过利用Java框架为大规模数据处理提供了一个平台无关的，高抽象的编程范式。尽管Java提供了跨各种机器的软件可移植性，但它也限制了分布式环境（如Spark）的性能。虽然用更快的语言重写像Spark这样的平台可能是不现实的，但一种更可行的方法来缓解其糟糕的性能是加速计算，同时仍然在基于Java的框架内工作。本文演示了将FPGA加速集成到Spark中的可行性，使用MapReduce实现的k-means聚类算法来表明，即使使用性能没有很好优化的硬件平台，加速也是可能的。我们的方法的一个重要特征是，通过使用库函数，FPGA的使用对用户完全透明，这是用户访问Spark提供的函数的常见方式。高级用户可以使用高级合成进一步开发其他计算。
Apache Spark has become one of the most popular engines for big data processing. Spark provides a platform-independent, high-abstraction programming paradigm for large-scale data processing by leveraging the Java framework. Though it provides software portability across various machines, Java also limits the performance of distributed environments, such as Spark. While it may be unrealistic to rewrite platforms like Spark in a faster language, a more viable approach to mitigate its poor performance is to accelerate the computations while still working within the Java-based framework. This paper demonstrates the feasibility of incorporating FPGA acceleration into Spark,, uses a MapReduce implementation of the k-means clustering algorithm to show that acceleration is possible even when using a hardware platform that is not well-optimized for performance. An important feature of our approach is that the use of FPGAs is completely transparent to the user through the use of library functions, which is a common way by which users access functions provided by Spark. Power users can further develop other computations using high-level synthesis.
影响因子：
4.6
作者：
J. Xu
通讯作者：
J. Xu