SHF: Small: Hyperscaling Data Analytics for High-Performance Computers

数据分析通常在机器学习技术的帮助下，从大量数据集中提取见解。该项目的目标是允许领域专家，包括数据科学家，使用世界上最强大的超级计算系统快速分析大量数据集。问题是，最先进的数据处理算法过滤数据、总结结果和组合来自不同来源的信息，这些算法具有固有的可伸缩性瓶颈。该项目设计了超可扩展的数据处理算法，利用高性能计算机前所未有的计算、存储和网络并发性。该项目还开发了一个开源数据处理引擎，向公众传播这些算法的原型实现。另一个贡献是为本科生数据科学课程创建了大规模并行数据处理模块和相关教材，例如俄亥俄州立大学的数据分析本科专业。在现代数据中心中，极端计算并行性、快速网络和不断增长的内存容量的融合为为仓库级计算机设计超可扩展的数据处理内核提供了机会。这个项目位于数据管理和高性能计算的交叉点；它开发了可扩展的连接和聚合算法，拓扑感知查询规划和优化技术，以及用于共享冷存储的干扰感知数据访问方法。这是通过仔细重叠通信和计算、识别和避免不可伸缩的全对全通信、考虑网络路径拥塞和远程内存访问延迟中的可变性、明智地使用进程间协调来加速从大规模并行共享文件系统中摄取数据来实现的。这些研究活动为数据分析在仓库级计算机上的可扩展性和效率奠定了智力基础。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data analytics extracts insights from massive datasets, often with the assistance of machine learning techniques. The goal of this project is to allow domain experts, including data scientists, to analyze massive datasets quickly using the most powerful supercomputing systems in the world. The problem is that state-of-the-art data processing algorithms that filter data, summarize results and combine information from different sources have inherent scalability bottlenecks. This project designs hyperscalable data processing algorithms that harness the unprecedented compute, storage and networking concurrency of a high-performance computer. This project also develops an open-source data processing engine to disseminate prototype implementations of these algorithms to the public. Another contribution is the creation of a massively parallel data processing module and associated teaching materials for undergraduate data science curricula, such as the diverse Data Analytics undergraduate major at The Ohio State University.The confluence of extreme compute parallelism, fast networking and growing memory capacities in the modern datacenter presents an opportunity to design a hyperscalable data processing kernel for warehouse-scale computers. This project sits at the intersection of data management and high-performance computing; it develops scalable join and aggregation algorithms, topology-conscious query planning and optimization techniques, and interference-aware data access methods for shared cold storage. This is accomplished by carefully overlapping communication and computation, identifying and avoiding unscalable all-to-all communication, accounting for network path congestion and variability in remote memory access latency, and judiciously using inter-process coordination to accelerate data ingestion from a massively parallel shared file system. These research activities lay the intellectual foundation to make data analytics scalable and efficient in warehouse-scale computers.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.