EAGER:High Performance Algorithms for Interactive Data Science at Scale

EAGER：大规模交互式数据科学的高性能算法

• 批准号: 2109988
• 负责人: David Bader
• 金额: $ 18.74万
• 依托单位: New Jersey Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-03-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2109988&HistoricalAwards=false
• 关键词: EAGER; Performance; Algorithms; Interactive; Data

A real-world challenge in data science is to develop interactive methods for quickly analyzing new and novel data sets that are potentially of massive scale. This award will design and implement fundamental algorithms for high performance computing solutions that enable the interactive large-scale data analysis of massive data sets. Based on the widely-used data types and structures of strings, sets, matrices and graphs, this methodology will produce efficient and scalable software for three classes of fundamental algorithms that will drastically improve the performance on a wide range of real-world queries or directly realize frequent queries. These innovations will allow the broad community to move massive-scale data exploration from time-consuming batch processing to interactive analyses that give a data analyst the ability to comprehensively, deeply and efficiently explore the insights and science in real world data sets. By enabling the increasing number of developers to easily manipulate large data sets, this will greatly enlarge the data science community and find much broader use in new communities. Materials from this project will be included in graduate and undergraduate course curriculum. Especially, women, high school students and other underrepresented groups in STEM areas will be encouraged to participate in this research activity. This project focuses on these three important data structures for data analytics: 1) suffix array construction, 2) 'treap' construction and 3) distributed memory join algorithms, useful for analyzing large scale strings, implementing random search in large string data sets, and generating new relations, respectively. These fundamental algorithms serve as the cornerstone to support interactive data science at scale. Based on the theoretical achievements and systematic algorithm design, a novel symbiotic optimization methodology that can combine the theoretical analysis, data structure features, and typical data distribution features together as a whole will be developed to significantly improve the practical performance of the proposed algorithms. To evaluate and show the effectiveness of the proposed algorithms, these algorithms will be implemented in and contribute to an open source NumPy-like software framework that aims to provide productive data discovery tools on massive, dozens-of-terabytes data sets by bringing together the productivity of Python with world-class high performance computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

数据科学中一个现实世界的挑战是开发交互式方法来快速分析潜在大规模的新的和新颖的数据集。该奖项将设计和实施高性能计算解决方案的基本算法，使海量数据集的交互式大规模数据分析成为可能。基于广泛使用的字符串、集合、矩阵和图形的数据类型和结构，该方法将为三类基本算法生成高效和可扩展的软件，这些软件将显著提高对广泛现实世界查询的性能或直接实现频繁查询。这些创新将使广大社区能够将大规模数据探索从耗时的批处理转移到交互式分析，使数据分析师能够全面、深入和高效地探索现实世界数据集中的洞察力和科学。通过使越来越多的开发人员能够轻松地操作大型数据集，这将极大地扩大数据科学界，并在新的社区中找到更广泛的用途。这个项目的材料将包括在研究生和本科课程的课程中。特别是，将鼓励STEM地区的妇女、高中生和其他代表性不足的群体参加这项研究活动。本项目主要研究数据分析中的三种重要数据结构：1)后缀数组构造，2)‘treap’构造和3)分布式内存连接算法，分别用于分析大规模字符串、在大型字符串数据集中实现随机搜索和生成新的关系。这些基本算法是支持大规模交互数据科学的基石。在理论成果和系统算法设计的基础上，提出一种新的共生优化方法，将理论分析、数据结构特征和典型数据分布特征有机地结合在一起，显著提高算法的实用性能。为了评估和展示提出的算法的有效性，这些算法将在一个类似NumPy的开源软件框架中实现并做出贡献，该框架旨在通过将Python的生产力与世界一流的高性能计算相结合，在海量、数十TB的数据集上提供高效的数据发现工具。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Triangle Counting Through Cover-Edges

通过盖边缘进行三角形计数

• 发表时间: 2023
• 期刊: The 27th Annual IEEE High Performance Extreme Computing Conference (HPEC
• 作者: Bader, David;Li, Fuhuan;Ganeshan, Anya;Gundogdu, Ahmet;Lew, Jason;Alvarado Rodriguez, Oliver;Du, Zhihui
• 通讯作者: Du, Zhihui

Triangle Centrality in Arkouda

Arkouda 的三角形中心性

• 发表时间: 2022
• 期刊: The 26th Annual IEEE High Performance Extreme Computing Conference (HPEC
• 作者: Joseph Patchett;Zhihui Du;Fuhuan Li;David A. Bader
• 通讯作者: David A. Bader

Parallel Longest Common SubSequence Analysis In Chapel

• DOI: 10.1109/hpec58863.2023.10363472
• 发表时间: 2023-09
• 期刊: 2023 IEEE High Performance Extreme Computing Conference (HPEC)
• 作者: Soroush Vahidi;B. Schieber;Zhihui Du;David A. Bader

Anti-Section Transitive Closure

• DOI: 10.1109/hipc53243.2021.00033
• 发表时间: 2021-12
• 期刊: 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)
• 作者: Oded Green;Zhihui Du;Sanyamee Patel;Zehui Xie;Hang Liu;David A. Bader

Fast Triangle Counting

快速三角形计数

• 发表时间: 2023
• 期刊: The 27th Annual IEEE High Performance Extreme Computing Conference (HPEC
• 作者: Bader, David