Collaborative Research: Hardware-Aware Matrix Computations for Deep Learning Applications

如今，深度学习（DL）系统无处不在，可以说比任何其他计算系统都更能影响我们的日常生活。最近，这种深度模型（例如，GPT-3）变得越来越大且笨重，具有大的计算占用空间。考虑到不断增长的计算需求，在一些大型技术公司之外，在学习这种DL模型的前沿研究方面取得进展几乎是不可能的。这个项目将探索原则性的方法来创建深度学习系统，这些系统与大型深度模型一样具有表现力，但计算成本只有一小部分。在实践方面，这些改进有望将创建这种强大的DL模型的可能性扩展到社会的更大部分。在教育方面，该项目将培训本科生（UG）研究人员，并将责任计算纳入UG课程。该项目将研究如何使用结构化矩阵与现代硬件约束相结合，以实现与这些真正的大型模型相似的性能，但尺寸和计算成本却只有一小部分。具体来说，研究人员专注于以下两个方面：（i）设计满足DL应用中所需的所有属性的结构化矩阵的“圣杯”（包括具有高效的投影问题以及具有高效的并行和/或硬件友好的学习算法）;以及（ii）考虑我们的新理论可以解锁的新应用。这种DL透镜暴露了研究结构矩阵时需要考虑的新问题。反过来，在这个项目中研究的结构矩阵的新家庭不仅将有直接的实际应用，但也将解开矩阵计算中的经典理论问题的新的转折。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Deep Learning (DL) systems these days are ubiquitous, and arguably affect our everyday lives more than any other computational system. Recently, such deep models (e.g., GPT-3) have increasingly become large and unwieldy with a large computational footprint. Given the ever increasing computational requirements, it has become nearly impossible to make progress on cutting edge research in learning such DL models outside of a few large technological companies. This project will explore principled ways to create DL systems that are as expressive as the large deep models but at a fraction of the computational cost. On the practical front these improvements are expected to expand the possibility of creating such powerful DL models to larger parts of society. On the educational front, this project will train undergraduate (UG) researchers and will integrate responsible computing into UG curriculum.This project will study how one can use structured matrices in concert with modern hardware constraints to achieve similar performance as these really large models but at a fraction of size and computational cost. Specifically, the investigators focus on the following two thrusts: (i) Design the ‘holy grail’ of structured matrices that satisfy all properties that are desirable in DL applications (including having an efficient projection problem as well as having efficient parallel and/or hardware friendly learning algorithms); and (ii) Thinking of new applications that our new theory can unlock. This DL lens exposes new problems to consider when studying structured matrices. In turn, the new family of structured matrices studied in this project will not only have immediate practical applications but will also unlock new twists on classical theoretical problems in matrix computations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.