Collaborative Research: PPoSS: Planning: Cross-layer Coordination and Optimization for Scalable and Sparse Tensor Networks (CROSS)

合作研究：PPoSS：规划：可扩展和稀疏张量网络的跨层协调和优化（CROSS）

• 批准号: 2217086
• 负责人: Dong Li
• 金额: $ 6.25万
• 依托单位: University of California - Merced
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2024-02-29
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2217086&HistoricalAwards=false
• 关键词: Collaborative; Research; PPoSS; Planning; Cross

High-dimensional data computation or analytics are gaining importance in many domains, such as quantum chemistry/physics, quantum circuit simulation, brain processing, social networks, healthcare and machine/deep learning, to name a few. Tensors, a representation of high-dimensional data, are playing an increasingly critical role, and so are tensor methods. Tensor decompositions or factorizations of low-dimensional data (three to five dimensions) have been extensively studied over the past years from a high-performance computing and also compiler and computer architecture angles for their computational core operations, while tensor networks targeting very high-dimensional data (over ten dimensions) and extracting physically meaningful latent variables are underdeveloped because of their complicated mathematical nature, extremely high computational complexity, and more domain-dependent challenges. The project’s novelties are manifold: 1) memory heterogeneity-aware representations with algorithm and system optimizations, which could be adopted to solve other problems such as irregular applications and sparse numerical methods; 2) hardware-software co-design of specialized, sparse-tensor network-accelerator architectures, that are among the first hardware implementations of sparse-tensor networks. The project’s impacts are 1) advancing state-of-the-art tensor decomposition studies to model true higher-order and sparse data; 2) triggering a closer long-term collaboration ranging from academia to research labs to industry by studying solicitous applications; 3) bringing appropriate educational opportunities.This project proposes Cross-layer cooRdination and Optimization for Scalable and Sparse-Tensor Networks (CROSS) for heterogeneous systems that are equipped with various types of accelerators, such as GPUs, TPUs and FPGAs, as well as heterogeneous memories with dynamic and non-volatile random-access memories (DRAM+NVRAM). This research aims to study the sparsity in widely used tensor networks by introducing constraints, regularization, dictionaries, and/or domain knowledge for better data compression, faster computation, lower memory usage and better interpretability. Besides the sparsity challenges, sparse-tensor networks also suffer from the curse of dimensionality, aggravated data randomness and irregular program and memory access behaviors. This planning project conducts preliminary research that aims to address these challenges from four perspectives: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contraction (SpTC) algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse-tensor networks. The optimized sparse tensor networks will encompass efforts from high-performance computing, algorithms, compilers, computer architecture and performance modeling and will be tested under multiple application scenarios.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

高维数据计算或分析在许多领域中越来越重要，例如量子化学/物理学、量子电路模拟、大脑处理、社交网络、医疗保健和机器/深度学习等。张量是高维数据的一种表示，它正在发挥越来越重要的作用，张量方法也是如此。低维数据的张量分解或因子分解在过去的几年里，从高性能计算以及编译器和计算机架构的角度对其计算核心操作进行了广泛的研究，而针对非常高维数据的张量网络（超过十个维度）和提取物理上有意义的潜在变量是不发达的，因为它们复杂的数学性质，非常高的计算复杂性，以及更多的领域相关挑战。该项目的新颖性是多方面的：1）具有算法和系统优化的内存异构性感知表示，可用于解决其他问题，如不规则应用和稀疏数值方法; 2）专用的稀疏张量网络加速器架构的硬件-软件协同设计，这是稀疏张量网络的第一批硬件实现之一。该项目的影响是：1）推进最先进的张量分解研究，以模拟真正的高阶和稀疏数据; 2）通过研究热心的应用，引发从学术界到研究实验室再到工业界的更密切的长期合作; 3）提供适当的教育机会。本项目提出了可扩展和稀疏张量网络的跨层协调和优化（CROSS）适用于配备各种类型加速器的异构系统，例如GPU、TPU和FPGA，以及具有动态和非易失性随机存取存储器（DRAM+NVRAM）的异构存储器。本研究旨在通过引入约束、正则化、字典和/或领域知识来研究广泛使用的张量网络中的稀疏性，以实现更好的数据压缩、更快的计算、更低的内存使用和更好的可解释性。除了稀疏性的挑战，稀疏张量网络还遭受维数灾难，加剧了数据的随机性和不规则的程序和内存访问行为。该规划项目进行了初步研究，旨在从四个角度解决这些挑战：（1）内存异质性感知表示和数据（重新）排列，（2）具有智能页面排列的平衡稀疏张量收缩（SpTC）算法，（3）记忆化和智能分配以降低计算成本，以及（4）稀疏张量网络的专用加速器架构。优化的稀疏张量网络将包括高性能计算、算法、编译器、计算机架构和性能建模方面的工作，并将在多种应用场景下进行测试。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Merchandiser: Data Placement on Heterogeneous Memory for Task-Parallel HPC Applications with Load-Balance Awareness

• DOI: 10.1145/3572848.3577497
• 发表时间: 2023-02
• 期刊: Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming
• 作者: Zhen Xie;Jie Liu-;Jiajia Li;Dong Li