Collaborative Research: PPoSS: LARGE: Cross-layer Coordination and Optimization for Scalable and Sparse Tensor Networks (CROSS)

高维数据计算和分析在量子化学/物理、量子电路模拟、社交网络、医疗保健和机器/深度学习等各个领域越来越重要。张量，一种高维数据的表示，已经变得越来越重要。虽然广泛的研究集中在低维数据的分解和因子分解等张量方法上，但在满足高维数据（超过十维）并可以提取物理上有意义的潜在变量的张量网络方面，明显缺乏发展。这些挑战来自于它们复杂的数学性质、极高的计算复杂性和特定领域的困难。该项目旨在通过设计高效的张量网络来弥合这一关键差距，特别是对于稀疏数据，这在许多现实世界的应用中都很普遍。该项目的影响包括四个方面：1)改善张量网络的数据压缩、计算、内存使用和可解释性；2)在学术界、国家研究实验室和工业界之间建立持久的合作伙伴关系，共同关注上述应用；3)通过设计相关新课程、培训本科生和研究生、组织研讨会和加强K-12外展来拓宽教育途径。该项目深入研究了可扩展和稀疏张量网络（CROSS）的跨层协调和优化，该网络设计用于配备各种加速器的异构系统，如图形处理单元（gpu），张量处理单元（tpu）和现场可编程门阵列（fpga），以及各种存储器，如动态和非易失性随机存取存储器。本研究旨在通过结合约束、正则化、字典和领域知识来研究广泛使用的张量网络中的稀疏性。除了稀疏性方面的挑战，稀疏张量网络还面临维度、数据随机性加剧、程序和内存访问行为不规范等问题。本研究从四个方面解决了这些挑战：(1)内存异构感知表示和数据（重新）排列；(2)具有智能页面排列的平衡稀疏张量收缩算法；(3)记忆和智能分配以降低计算成本；(4)稀疏张量网络的专用加速器架构。优化的稀疏张量网络代表了高性能计算、算法、编译器、计算机体系结构和性能建模等专业知识的协同努力。提出的解决方案在不同的应用场景和广泛的硬件环境下进行评估，以证明其在现实环境中的有效性和适用性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

High-dimensional data computation and analytics are gaining importance in various domains, such as quantum chemistry/physics, quantum circuit simulation, social networks, healthcare, and machine/deep learning. Tensors, a representation of high-dimensional data, have become increasingly crucial. While extensive research has focused on tensor methods like decompositions and factorizations for low-dimensional data, there is a notable lack of development in tensor networks that cater to high-dimensional data (over ten dimensions) and can extract physically meaningful latent variables. The challenges arise from their complicated mathematical nature, extremely high computational complexity, and domain-specific difficulties. This project aims to bridge this critical gap by devising efficient tensor networks, especially for sparse data, which are prevalent in many real-world applications. The impacts of the project encompass four aspects: 1) Improving data compression, computation, memory usage, and interpretability of tensor networks; 2) fostering enduring and collaborative partnerships among academia, national research labs, and industry with a shared focus on the aforementioned applications; and 3) broadening education avenues by designing relevant new courses, training undergraduate and graduate students, organizing workshops, and enhancing K-12 outreach. This project delves into Cross-layer cooRdination and Optimization for Scalable and Sparse Tensor Networks (CROSS) designed for heterogeneous systems equipped with diverse accelerators like Graphics Processing Units (GPUs), Tensor Processing Units (TPUs) and Field Programmable Gate Arrays (FPGAs), and various memories such as dynamic and non-volatile random-access memories. This research aims to study sparsity within widely used tensor networks by incorporating constraints, regularization, dictionary, and domain knowledge. In addition to sparsity challenges, sparse tensor networks also face problems such as dimensionality, exacerbated data randomness and irregular program and memory access behaviors. This research tackles these challenges from four dimensions: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contraction algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse tensor networks. The optimized sparse tensor networks represent a synergistic effort combining expertise from high-performance computing, algorithms, compilers, computer architecture and performance modeling. The proposed solutions are evaluated under diverse application scenarios and across a wide range of hardware environments to demonstrate their effectiveness and applicability in real-world settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.