BIGDATA: F: DKM: Collaborative Research: Scalable Middleware for Managing and Processing Big Data on Next Generation HPC Systems

管理和处理海量数据并获得有意义的见解是大数据社区面临的重大挑战。因此，精心设计处理此类数据的数据密集型计算中间件(如Hadoop、HBase和Spark)，具有高性能和可扩展性，以满足此类大数据应用日益增长的需求，是至关重要的。虽然Hadoop、Spark和HBase在处理大数据应用程序方面越来越受欢迎，但这些中间件和相关的大数据应用程序无法利用在世界各地广泛部署的现代高性能计算(HPC)系统上的高级功能，包括XSEDE环境中的许多多千万亿次浮点系统。在过去的十年中，现代HPC系统和相关的中间件(如MPI和并行文件系统)一直在利用HPC技术(多核/多核体系结构、支持RDMA的网络、NVRAM和固态硬盘)的进步。然而，大数据中间件(如Hadoop、HBase和Spark)并未采用此类技术。这些差异正在将HPC和大数据处理带入“不同的轨道”。这项拟议的研究由俄亥俄州立大学和SDSC的计算机和应用科学家组成的团队进行，旨在将高性能计算和大数据处理带入“融合的轨道”。调查人员将具体解决以下挑战：1)为大数据处理设计新颖的通信和I/O运行时，同时利用现代多/多核、网络和存储技术的特征；2)重新设计大数据中间件(如Hadoop、HBase和Spark)，以在现代和下一代HPC系统上提供性能和可扩展性；以及3)演示建议方法对HPC系统上一系列驱动大数据应用程序的好处。拟议工作的目标是使用流行的大数据中间件(Hadoop、HBase和Spark)的大数据社区中的四个主要工作负载和应用程序(即数据分析、查询、交互和迭代)。建议的框架将在各种大数据基准和应用程序上进行验证。提议的中间件和运行时将向社区公开提供。这项研究通过对俄亥俄州立大学和SDSC新的数据分析项目中关键课程的教育学研究，推动了课程的进步--这是全国首批此类课程之一。

Managing and processing large volumes of data and gaining meaningful insights is a significant challenge facing the Big Data community. Thus, it is critical that data-intensive computing middleware (such as Hadoop, HBase and Spark) to process such data are diligently designed, with high performance and scalability, in order to meet the growing demands of such Big Data applications. While Hadoop, Spark and HBase are gaining popularity for processing Big Data applications, these middleware and the associated Big Data applications are not able to take advantage of the advanced features on modern High Performance Computing (HPC) systems widely deployed all over the world, including many of of the multi-Petaflop systems in the XSEDE environment. Modern HPC systems and the associated middleware (such as MPI and Parallel File systems) have been exploiting the advances in HPC technologies (multi/many-core architectures, RDMA-enabled networking, NVRAMs and SSDs) during the last decade. However, Big Data middleware (such as Hadoop, HBase and Spark) have not embraced such technologies. These disparities are taking HPC and Big Data processing into "divergent trajectories." The proposed research, undertaken by a team of computer and application scientists from OSU and SDSC, aim to bring HPC and Big Data processing into a "convergent trajectory." The investigators will specifically address the following challenges: 1) designing novel communication and I/O runtime for Big Data processing while exploiting the features of modern multi-/many-core, networking and storage technologies; 2) redesigning Big Data middleware (such as Hadoop, HBase and Spark) to deliver performance and scalability on modern and next-generation HPC systems; and 3) demonstrating the benefits of the proposed approach for a set of driving Big Data applications on HPC system. The proposed work targets four major workloads and applications in the Big Data community (namely data analytics, query, interactive, and iterative) using the popular Big Data middleware (Hadoop, HBase and Spark). The proposed framework will be validated on a variety of Big Data benchmarks and applications. The proposed middleware and runtimes will be made publicly available to the community. The research enables curricular advancements via research in pedagogy for key courses in the new data analytics program at Ohio State and SDSC -- among the first of its kind nationwide.