SPIDAL Java: high performance data analytics with Java and MPI on large multicore HPC clusters
SPIDAL Java: high performance data analytics with Java and MPI on large multicore HPC clusters
复制标题
SPIDAL Java：在大型多核 HPC 集群上使用 Java 和 MPI 进行高性能数据分析
DOI：
--
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
G. Fox
中科院分区：
文献类型：
--
作者：
S. Ekanayake;Supun Kamburugamuve;G. Fox
在过去几年中，基于 Java 的大数据框架和库（例如 Apache Hadoop、Spark 和 Storm）做出了重大贡献。虽然这些系统具有丰富的互操作性和功能，但开发高性能大数据分析应用程序仍具有挑战性。此外，这些应用的文献中缺乏对性能特征和高性能优化的研究。相比之下，这些功能在高性能计算 (HPC) 领域中有详细记录，并且某些技术在大数据领域也具有潜在的性能优势。本文介绍了高性能大数据分析库 - SPIDAL Java 的实现，并全面讨论了五个性能挑战、解决方案和加速结果。 SPIDAL Java 捕获了一类具有大量计算和通信功能的全局机器学习应用程序，可以作为使用 Java 大数据分析研究性能瓶颈的标准。这里提出的五个挑战是节点内消息传递的成本、低效的缓存利用率、线程的性能成本、垃圾收集的开销以及堆分配对象的成本。 SPIDAL Java 针对这些问题提出了解决方案，并在最新的基于 Intel Haswell 的多核集群之一中运行多达 3072 个核心时展示了显着的性能提升和可扩展性。
Within the last few years, there have been significant contributions to Java-based big data frameworks and libraries such as Apache Hadoop, Spark, and Storm. While these systems are rich in interoperability and features, developing high performance big data analytic applications is challenging. Also, the study of performance characteristics and high performance optimizations is lacking in the literature for these applications. By contrast, these features are well documented in the High Performance Computing (HPC) domain and some of the techniques have potential performance benefits in the big data domain as well. This paper presents the implementation of a high performance big data analytics library - SPIDAL Java - with a comprehensive discussion on five performance challenges, solutions, and speedup results. SPIDAL Java captures a class of global machine learning applications with significant computation and communication that can serve as a yardstick in studying performance bottlenecks with Java big data analytics. The five challenges present here are the cost of intra-node messaging, inefficient cache utilization, performance costs with threads, overhead of garbage collection, and the costs of heap allocated objects. SPIDAL Java presents its solutions to these and demonstrates significant performance gains and scalability when running on up to 3072 cores in one of the latest Intel Haswell-based multicore clusters.