CIF21 DIBBs: Middleware and High Performance Analytics Libraries for Scalable Data Science

许多科学问题依赖于对大量数据的分析和计算能力。这种分析通常不能很好地扩展；它的有效性受到大数据日益增长的数量、种类和变化速度（速度）的阻碍。该项目将设计、开发和实施构建模块，从根本上提高在广泛的网络基础设施上支持数据密集型分析的能力，包括NSF为科学界支持的基础设施。该项目将集成传统高性能计算的功能，如科学图书馆、通信和资源管理中间件，以及商业大数据生态系统中的丰富功能。后者包括许多重要的软件系统，如Hadoop，可以从Apache开源社区获得。亚利桑那大学、埃默里大学、印第安纳大学（领导）、堪萨斯大学、罗格斯大学、弗吉尼亚理工大学和犹他大学的大学团队之间的合作提供了设计和成功执行该项目所需的广泛专业知识。该项目将吸引科学家和教育工作者参加年度研讨会，并在特定学科的会议上开展活动，以收集对其软件的需求和反馈。它将包括具有夏季经验的代表性不足的社区，并将开发课程模块，其中包括构建为“数据分析即服务”的演示。该项目将为数据密集型分析和科学（MIDAS）设计并实现一个软件中间件，使应用程序具有高性能计算（HPC）的性能和Apache大数据栈的丰富功能。此外，本项目将设计并实现一套横切的高性能数据分析库；SPIDAL（可扩展并行互操作数据分析库）将支持新的编程和执行模型，用于广泛的科学和工程应用中的数据密集型分析。该项目解决了七个不同领域的主要数据挑战：生物分子模拟、网络和计算社会科学、流行病学、计算机视觉、空间地理信息系统、极地科学遥感和病理信息学。项目库将对数据分析产生与PETSc、MPI和ScaLAPACK等科学库对超级计算机模拟同样有益的影响。这些库将在包括云、集群和超级计算机在内的一系列计算系统中实现可扩展和可互操作。

Many scientific problems depend on the ability to analyze and compute on large amounts of data. This analysis often does not scale well; its effectiveness is hampered by the increasing volume, variety and rate of change (velocity) of big data. This project will design, develop and implement building blocks that enable a fundamental improvement in the ability to support data intensive analysis on a broad range of cyberinfrastructure, including that supported by NSF for the scientific community. The project will integrate features of traditional high-performance computing, such as scientific libraries, communication and resource management middleware, with the rich set of capabilities found in the commercial Big Data ecosystem. The latter includes many important software systems such as Hadoop, available from the Apache open source community. A collaboration between university teams at Arizona, Emory, Indiana (lead), Kansas, Rutgers, Virginia Tech, and Utah provides the broad expertise needed to design and successfully execute the project. The project will engage scientists and educators with annual workshops and activities at discipline-specific meetings, both to gather requirements for and feedback on its software. It will include under-represented communities with summer experiences, and will develop curriculum modules that include demonstrations built as 'Data Analytics as a Service.'The project will design and implement a software Middleware for Data-Intensive Analytics and Science (MIDAS) that will enable scalable applications with the performance of HPC (High Performance Computing) and the rich functionality of the commodity Apache Big Data Stack. Further, this project will design and implement a set of cross-cutting high-performance data-analysis libraries; SPIDAL (Scalable Parallel Interoperable Data Analytics Library) will support new programming and execution models for data-intensive analysis in a wide range of science and engineering applications. The project addresses major data challenges in seven different communities: Biomolecular Simulations, Network and Computational Social Science, Epidemiology, Computer Vision, Spatial Geographical Information Systems, Remote Sensing for Polar Science, and Pathology Informatics. The project libraries will have the same beneficial impact on data analytics that scientific libraries such as PETSc, MPI and ScaLAPACK have had for supercomputer simulations. These libraries will be implemented to be scalable and interoperable across a range of computing systems including clouds, clusters and supercomputers.