Collaborative Research: PPoSS: LARGE: Cross-layer Coordination and Optimization for Scalable and Sparse Tensor Networks (CROSS)

合作研究：PPoSS：LARGE：可扩展和稀疏张量网络的跨层协调和优化（CROSS）

• 批准号: 2316201
• 负责人: Jiajia Li
• 金额: $ 303.38万
• 依托单位: North Carolina State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-15 至 2028-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2316201&HistoricalAwards=false
• 关键词: Collaborative; Research; PPoSS; LARGE; Cross

High-dimensional data computation and analytics are gaining importance in various domains, such as quantum chemistry/physics, quantum circuit simulation, social networks, healthcare, and machine/deep learning. Tensors, a representation of high-dimensional data, have become increasingly crucial. While extensive research has focused on tensor methods like decompositions and factorizations for low-dimensional data, there is a notable lack of development in tensor networks that cater to high-dimensional data (over ten dimensions) and can extract physically meaningful latent variables. The challenges arise from their complicated mathematical nature, extremely high computational complexity, and domain-specific difficulties. This project aims to bridge this critical gap by devising efficient tensor networks, especially for sparse data, which are prevalent in many real-world applications. The impacts of the project encompass four aspects: 1) Improving data compression, computation, memory usage, and interpretability of tensor networks; 2) fostering enduring and collaborative partnerships among academia, national research labs, and industry with a shared focus on the aforementioned applications; and 3) broadening education avenues by designing relevant new courses, training undergraduate and graduate students, organizing workshops, and enhancing K-12 outreach. This project delves into Cross-layer cooRdination and Optimization for Scalable and Sparse Tensor Networks (CROSS) designed for heterogeneous systems equipped with diverse accelerators like Graphics Processing Units (GPUs), Tensor Processing Units (TPUs) and Field Programmable Gate Arrays (FPGAs), and various memories such as dynamic and non-volatile random-access memories. This research aims to study sparsity within widely used tensor networks by incorporating constraints, regularization, dictionary, and domain knowledge. In addition to sparsity challenges, sparse tensor networks also face problems such as dimensionality, exacerbated data randomness and irregular program and memory access behaviors. This research tackles these challenges from four dimensions: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contraction algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse tensor networks. The optimized sparse tensor networks represent a synergistic effort combining expertise from high-performance computing, algorithms, compilers, computer architecture and performance modeling. The proposed solutions are evaluated under diverse application scenarios and across a wide range of hardware environments to demonstrate their effectiveness and applicability in real-world settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

高维数据计算和分析在各种领域都变得重要，例如量子化学/物理，量子电路模拟，社交网络，医疗保健以及机器/深度学习。张量是高维数据的表示，越来越重要。尽管广泛的研究集中在张量方法上，例如分解和低维数据的因素化，但在张量网络中显着缺乏迎合高维数据（超过十个维度）的发展，并且可以提取物理意义的潜在变量。挑战是由于它们复杂的数学性质，极高的计算复杂性和特定领域的困难。该项目旨在通过设计有效的张量网络来弥合这一关键差距，尤其是对于稀疏数据，这些数据在许多现实世界中都普遍存在。项目的影响包括四个方面：1）改善张量网络的数据压缩，计算，内存使用和可解释性； 2）在学术界，国家研究实验室和行业之间建立持久和协作伙伴关系，共同关注上述应用程序； 3）通过设计相关的新课程，培训本科生和研究生，组织研讨会并增强K-12外展活动来扩大教育途径。该项目深入研究了跨层协调和优化，用于可扩展和稀疏的张量网络（Cross），专为配备有多种加速器的异质系统设计，例如图形处理单元（GPU）（GPU），张量处理单元（TPU）（TPU）和现场可编程的门阵列和可动态的录音，以及各种记忆，以及各种记忆。这项研究旨在通过结合约束，正则化，字典和域知识来研究广泛使用的张量网络中的稀疏性。除了稀疏挑战外，稀疏张量网络还面临诸如维度，加剧数据随机性以及不规则的程序和内存访问行为等问题。这项研究从四个方面解决了这些挑战：（1）内存异质性意识到的表示和数据（重新）安排，（2）具有智能页面布置的均衡稀疏稀疏张量收缩算法，（3）记忆和智能分配，以减少计算成本，以及（4）专用Accelerator Archator Archator ArchiteTures For Sparse Tensor网络网络的特殊Accelator ArchiteTures。优化的稀疏张量网络代表了一项协同努力，结合了高性能计算，算法，编译器，计算机体系结构和性能建模的专业知识。在各种应用程序方面和广泛的硬件环境中对所提出的解决方案进行了评估，以证明其在现实环境中的有效性和适用性。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子优点和更广泛的审查标准通过评估来评估的。