BIGDATA: F: DKM: Collaborative Research: Scalable Middleware for Managing and Processing Big Data on Next Generation HPC Systems

BIGDATA：F：DKM：协作研究：用于在下一代 HPC 系统上管理和处理大数据的可扩展中间件

• 批准号: 1447804
• 负责人: Dhabaleswar Panda
• 金额: $ 72.02万
• 依托单位: Ohio State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-09-01 至 2017-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1447804&HistoricalAwards=false
• 关键词: BIGDATA; DKM; Collaborative; Research; Scalable

Managing and processing large volumes of data and gaining meaningful insights is a significant challenge facing the Big Data community. Thus, it is critical that data-intensive computing middleware (such as Hadoop, HBase and Spark) to process such data are diligently designed, with high performance and scalability, in order to meet the growing demands of such Big Data applications. While Hadoop, Spark and HBase are gaining popularity for processing Big Data applications, these middleware and the associated Big Data applications are not able to take advantage of the advanced features on modern High Performance Computing (HPC) systems widely deployed all over the world, including many of of the multi-Petaflop systems in the XSEDE environment. Modern HPC systems and the associated middleware (such as MPI and Parallel File systems) have been exploiting the advances in HPC technologies (multi/many-core architectures, RDMA-enabled networking, NVRAMs and SSDs) during the last decade. However, Big Data middleware (such as Hadoop, HBase and Spark) have not embraced such technologies. These disparities are taking HPC and Big Data processing into "divergent trajectories." The proposed research, undertaken by a team of computer and application scientists from OSU and SDSC, aim to bring HPC and Big Data processing into a "convergent trajectory." The investigators will specifically address the following challenges: 1) designing novel communication and I/O runtime for Big Data processing while exploiting the features of modern multi-/many-core, networking and storage technologies; 2) redesigning Big Data middleware (such as Hadoop, HBase and Spark) to deliver performance and scalability on modern and next-generation HPC systems; and 3) demonstrating the benefits of the proposed approach for a set of driving Big Data applications on HPC system. The proposed work targets four major workloads and applications in the Big Data community (namely data analytics, query, interactive, and iterative) using the popular Big Data middleware (Hadoop, HBase and Spark). The proposed framework will be validated on a variety of Big Data benchmarks and applications. The proposed middleware and runtimes will be made publicly available to the community. The research enables curricular advancements via research in pedagogy for key courses in the new data analytics program at Ohio State and SDSC -- among the first of its kind nationwide.

管理和处理大量数据并获得有意义的见解是大数据社区面临的重大挑战。 因此，至关重要的是，用于处理这些数据的数据密集型计算中间件（如Hadoop，HBase和Spark）必须经过精心设计，具有高性能和可扩展性，以满足此类大数据应用程序不断增长的需求。 虽然Hadoop、Spark和HBase在处理大数据应用程序方面越来越受欢迎，但这些中间件和相关的大数据应用程序无法利用世界各地广泛部署的现代高性能计算（HPC）系统的高级功能，包括XSEDE环境中的许多多Petaflop系统。 在过去十年中，现代HPC系统和相关的中间件（如MPI和并行文件系统）一直在利用HPC技术（多/众核架构、支持RDMA的网络、NVRAM和SSD）的进步。然而，大数据中间件（如Hadoop，HBase和Spark）尚未采用此类技术。这些差异正在将HPC和大数据处理带入“不同的轨迹”。这项由来自俄勒冈州立大学和SDSC的计算机和应用科学家组成的团队进行的研究旨在将HPC和大数据处理带入“融合轨道”。“研究人员将专门解决以下挑战：1）为大数据处理设计新颖的通信和I/O运行时，同时利用现代多核/众核、网络和存储技术的功能; 2）重新设计大数据中间件（如Hadoop、HBase和Spark），在现代和下一代HPC系统上提供性能和可扩展性;以及3）展示了所提出的方法对于HPC系统上的一组驱动大数据应用的益处。 拟议的工作针对大数据社区中使用流行的大数据中间件（Hadoop、HBase和Spark）的四个主要工作负载和应用程序（即数据分析、查询、交互和迭代）。 拟议的框架将在各种大数据基准和应用程序上进行验证。 建议的中间件和运行时将公开提供给社区。 这项研究通过对俄亥俄州和SDSC新数据分析项目中关键课程的教学法研究，实现了课程的进步，这是全国同类项目中的第一个。