Collaborative Research: PPoSS: LARGE: Cross-layer Coordination and Optimization for Scalable and Sparse Tensor Networks (CROSS)

高维数据计算和分析在各个领域中越来越重要，例如量子化学/物理学、量子电路模拟、社交网络、医疗保健和机器/深度学习。张量是高维数据的一种表示，它变得越来越重要。虽然广泛的研究集中在张量方法上，如低维数据的分解和因子分解，但在满足高维数据（超过10维）并可以提取物理上有意义的潜在变量的张量网络中，明显缺乏发展。这些挑战来自于它们复杂的数学性质、极高的计算复杂性和特定领域的困难。该项目旨在通过设计高效的张量网络来弥合这一关键差距，特别是对于稀疏数据，这在许多现实世界的应用中很普遍。该项目的影响包括四个方面：1）改善张量网络的数据压缩，计算，内存使用和可解释性; 2）促进学术界，国家研究实验室和工业界之间的持久合作伙伴关系，共同关注上述应用;三是拓宽教育渠道，开设相关新课程，培养本科生和研究生，举办讲习班，并加强K-12的推广。该项目深入研究了可扩展和稀疏张量网络（CROSS）的跨层协调和优化，该网络专为配备各种加速器（如图形处理单元（GPU），张量处理单元（TPU）和现场可编程门阵列（FPGA））以及各种存储器（如动态和非易失性随机存取存储器）的异构系统而设计。本研究旨在通过结合约束、正则化、字典和领域知识来研究广泛使用的张量网络中的稀疏性。除了稀疏性的挑战，稀疏张量网络还面临着诸如维数、加剧的数据随机性以及不规则的程序和内存访问行为等问题。这项研究从四个维度解决了这些挑战：（1）内存异构感知表示和数据（重新）排列，（2）平衡稀疏张量收缩算法与智能页面排列，（3）记忆和智能分配，以降低计算成本，以及（4）专门的加速器架构稀疏张量网络。优化的稀疏张量网络代表了一种协同努力，它结合了高性能计算、算法、编译器、计算机体系结构和性能建模的专业知识。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

High-dimensional data computation and analytics are gaining importance in various domains, such as quantum chemistry/physics, quantum circuit simulation, social networks, healthcare, and machine/deep learning. Tensors, a representation of high-dimensional data, have become increasingly crucial. While extensive research has focused on tensor methods like decompositions and factorizations for low-dimensional data, there is a notable lack of development in tensor networks that cater to high-dimensional data (over ten dimensions) and can extract physically meaningful latent variables. The challenges arise from their complicated mathematical nature, extremely high computational complexity, and domain-specific difficulties. This project aims to bridge this critical gap by devising efficient tensor networks, especially for sparse data, which are prevalent in many real-world applications. The impacts of the project encompass four aspects: 1) Improving data compression, computation, memory usage, and interpretability of tensor networks; 2) fostering enduring and collaborative partnerships among academia, national research labs, and industry with a shared focus on the aforementioned applications; and 3) broadening education avenues by designing relevant new courses, training undergraduate and graduate students, organizing workshops, and enhancing K-12 outreach. This project delves into Cross-layer cooRdination and Optimization for Scalable and Sparse Tensor Networks (CROSS) designed for heterogeneous systems equipped with diverse accelerators like Graphics Processing Units (GPUs), Tensor Processing Units (TPUs) and Field Programmable Gate Arrays (FPGAs), and various memories such as dynamic and non-volatile random-access memories. This research aims to study sparsity within widely used tensor networks by incorporating constraints, regularization, dictionary, and domain knowledge. In addition to sparsity challenges, sparse tensor networks also face problems such as dimensionality, exacerbated data randomness and irregular program and memory access behaviors. This research tackles these challenges from four dimensions: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contraction algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse tensor networks. The optimized sparse tensor networks represent a synergistic effort combining expertise from high-performance computing, algorithms, compilers, computer architecture and performance modeling. The proposed solutions are evaluated under diverse application scenarios and across a wide range of hardware environments to demonstrate their effectiveness and applicability in real-world settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.