BIGDATA: F: DKM: Collaborative Research: Scalable Middleware for Managing and Processing Big Data on Next Generation HPC Systems

管理和处理大量数据并获得有意义的见解是大数据社区面临的重大挑战。因此，精心设计数据密集型计算中间件（如Hadoop、HBase和Spark）来处理这些数据，具有高性能和可扩展性，以满足此类大数据应用日益增长的需求是至关重要的。虽然Hadoop、Spark和HBase在处理大数据应用方面越来越受欢迎，但这些中间件和相关的大数据应用无法利用世界各地广泛部署的现代高性能计算（HPC）系统的高级特性，包括XSEDE环境中的许多千万亿次浮点运算的系统。在过去的十年中，现代HPC系统和相关的中间件（如MPI和并行文件系统）一直在利用HPC技术的进步（多核/多核架构、支持rdma的网络、nvram和ssd）。然而，大数据中间件（如Hadoop、HBase和Spark）并没有采用这些技术。这些差异将HPC和大数据处理带入了“不同的轨迹”。这项提议的研究由俄勒冈州立大学和SDSC的计算机和应用科学家组成，旨在将HPC和大数据处理带入“融合轨迹”。研究人员将具体解决以下挑战：1)为大数据处理设计新颖的通信和I/O运行时，同时利用现代多核/多核、网络和存储技术的特点；2)重新设计大数据中间件（如Hadoop、HBase和Spark），以在现代和下一代HPC系统上提供性能和可扩展性；3)展示了所提出的方法对HPC系统上一系列驱动大数据应用的好处。建议的工作针对大数据社区中的四个主要工作负载和应用（即数据分析、查询、交互和迭代），使用流行的大数据中间件（Hadoop、HBase和Spark）。提出的框架将在各种大数据基准和应用程序上进行验证。提议的中间件和运行时将公开提供给社区。该研究通过对俄亥俄州立大学和SDSC新数据分析项目关键课程的教学法研究，使课程进步成为可能——这是全国首例。

Managing and processing large volumes of data and gaining meaningful insights is a significant challenge facing the Big Data community. Thus, it is critical that data-intensive computing middleware (such as Hadoop, HBase and Spark) to process such data are diligently designed, with high performance and scalability, in order to meet the growing demands of such Big Data applications. While Hadoop, Spark and HBase are gaining popularity for processing Big Data applications, these middleware and the associated Big Data applications are not able to take advantage of the advanced features on modern High Performance Computing (HPC) systems widely deployed all over the world, including many of of the multi-Petaflop systems in the XSEDE environment. Modern HPC systems and the associated middleware (such as MPI and Parallel File systems) have been exploiting the advances in HPC technologies (multi/many-core architectures, RDMA-enabled networking, NVRAMs and SSDs) during the last decade. However, Big Data middleware (such as Hadoop, HBase and Spark) have not embraced such technologies. These disparities are taking HPC and Big Data processing into "divergent trajectories." The proposed research, undertaken by a team of computer and application scientists from OSU and SDSC, aim to bring HPC and Big Data processing into a "convergent trajectory." The investigators will specifically address the following challenges: 1) designing novel communication and I/O runtime for Big Data processing while exploiting the features of modern multi-/many-core, networking and storage technologies; 2) redesigning Big Data middleware (such as Hadoop, HBase and Spark) to deliver performance and scalability on modern and next-generation HPC systems; and 3) demonstrating the benefits of the proposed approach for a set of driving Big Data applications on HPC system. The proposed work targets four major workloads and applications in the Big Data community (namely data analytics, query, interactive, and iterative) using the popular Big Data middleware (Hadoop, HBase and Spark). The proposed framework will be validated on a variety of Big Data benchmarks and applications. The proposed middleware and runtimes will be made publicly available to the community. The research enables curricular advancements via research in pedagogy for key courses in the new data analytics program at Ohio State and SDSC -- among the first of its kind nationwide.