Collaborative Research: PPoSS: LARGE: Cross-layer Coordination and Optimization for Scalable and Sparse Tensor Networks (CROSS)

高维数据计算和分析在量子化学/物理、量子电路模拟、社会网络、医疗保健和机器/深度学习等各个领域都变得越来越重要。张量是高维数据的一种表示形式，它变得越来越重要。虽然广泛的研究集中在张量方法，如低维数据的分解和因式分解，但张量网络明显缺乏发展，以迎合高维数据(超过10维)，并可以提取物理上有意义的潜在变量。这些挑战源于它们复杂的数学性质、极高的计算复杂性和特定于领域的困难。这个项目旨在通过设计有效的张量网络来弥合这一关键差距，特别是对于稀疏数据，这些数据在许多现实世界的应用中非常普遍。该项目的影响包括四个方面：1)改进张量网络的数据压缩、计算、内存使用和可解释性；2)促进学术界、国家研究实验室和产业界之间持久的合作伙伴关系，共同关注上述应用；以及3)通过设计相关的新课程、培训本科生和研究生、组织研讨会和加强K-12的推广，拓宽教育途径。该项目深入研究可扩展稀疏张量网络(CROSS)的跨层协调和优化，该网络专为配备了各种加速器(如图形处理单元(GPU)、张量处理单元(TPU)和现场可编程门阵列(FGA))以及各种存储器(如动态和非易失性随机存取存储器)的异类系统而设计。这项研究旨在通过结合约束、正则化、词典和领域知识来研究广泛使用的张量网络中的稀疏性。除了稀疏性的挑战，稀疏张量网络还面临着诸如维度、加剧的数据随机性和不规则的程序和存储器访问行为等问题。本研究从四个方面解决了这些挑战：(1)内存异构性感知的表示和数据(重)排列，(2)平衡稀疏张量压缩算法和智能页面排列，(3)记忆和智能分配以降低计算成本，以及(4)稀疏张量网络的专用加速器架构。优化的稀疏张量网络代表了高性能计算、算法、编译器、计算机体系结构和性能建模方面的专业知识的协同工作。建议的解决方案将在不同的应用场景和广泛的硬件环境下进行评估，以证明其在实际环境中的有效性和适用性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

High-dimensional data computation and analytics are gaining importance in various domains, such as quantum chemistry/physics, quantum circuit simulation, social networks, healthcare, and machine/deep learning. Tensors, a representation of high-dimensional data, have become increasingly crucial. While extensive research has focused on tensor methods like decompositions and factorizations for low-dimensional data, there is a notable lack of development in tensor networks that cater to high-dimensional data (over ten dimensions) and can extract physically meaningful latent variables. The challenges arise from their complicated mathematical nature, extremely high computational complexity, and domain-specific difficulties. This project aims to bridge this critical gap by devising efficient tensor networks, especially for sparse data, which are prevalent in many real-world applications. The impacts of the project encompass four aspects: 1) Improving data compression, computation, memory usage, and interpretability of tensor networks; 2) fostering enduring and collaborative partnerships among academia, national research labs, and industry with a shared focus on the aforementioned applications; and 3) broadening education avenues by designing relevant new courses, training undergraduate and graduate students, organizing workshops, and enhancing K-12 outreach. This project delves into Cross-layer cooRdination and Optimization for Scalable and Sparse Tensor Networks (CROSS) designed for heterogeneous systems equipped with diverse accelerators like Graphics Processing Units (GPUs), Tensor Processing Units (TPUs) and Field Programmable Gate Arrays (FPGAs), and various memories such as dynamic and non-volatile random-access memories. This research aims to study sparsity within widely used tensor networks by incorporating constraints, regularization, dictionary, and domain knowledge. In addition to sparsity challenges, sparse tensor networks also face problems such as dimensionality, exacerbated data randomness and irregular program and memory access behaviors. This research tackles these challenges from four dimensions: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contraction algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse tensor networks. The optimized sparse tensor networks represent a synergistic effort combining expertise from high-performance computing, algorithms, compilers, computer architecture and performance modeling. The proposed solutions are evaluated under diverse application scenarios and across a wide range of hardware environments to demonstrate their effectiveness and applicability in real-world settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.