Collaborative Research: SCALE MoDL: Representation Theoretic Foundations of Deep Learning

在过去的十年里，深度学习对整个社会产生了变革性的影响。然而，进步往往依赖于启发式方法、海量数据和强大的计算能力。这与有限的理论理解有关，并且有时会导致泛化失败和极端情况下的脆弱性能。该项目将通过使用表示理论为深度学习发展强大的理论基础来解决这些限制，表示理论是对称的数学研究。对称在人类推理中起着关键作用。对对称性在深度学习中扮演的角色的更深入的理解，将开启各种改进的模型。这些模型包括可以从科学知识中学习而不仅仅是从原始数据中学习的模型，具有可靠、有保证的性能的模型，以及可以重新组合他们已经学习的模式的模型——就像人类很容易做到的那样——以更快地推广到新情况。这个项目的一个明确目标是扩大对深度学习为何有效的研究。为此，研究人员将把研究与教育结合起来，并为来自科学领域代表性不足的群体的高中生建立一个导师计划。该研究的目标是了解表征理论在实现高效优化和改进深度学习泛化方面的作用，即使在具有近似或未知对称性的领域也是如此。该项目追求三条研究路线，将扩大表征理论在深度学习中的影响，超越严格的归纳偏差。首先是模型的对称度和域的对称度之间的权衡。这条研究路线将研究结合了等变和非等变特征的网络。研究的第二条线将检查直接从数据中学习对称性，以提高在没有已知对称性的领域中的泛化。第三个目标是为使用颤振表示的深度学习发展理论基础。这个视角通过它们的参数空间揭示了深度学习模型本身结构的对称性，即使这些域没有明显的对称性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In the past decade, deep learning has had transformative impacts across society. However, progress has often relied on heuristic methods, massive data, and great computing power. This comes with limited theoretical understanding and has at times given rise to failures of generalization and vulnerable performance in extreme scenarios. This project will address these limitations by developing strong theoretical foundations for deep learning using representation theory, which is the mathematical study of symmetry. Symmetry plays a key role in human reasoning. Greater understanding of the role symmetry plays in deep learning will unlock a variety of improved models. These include models that can learn from scientific knowledge and not just raw data, models with trustable, guaranteed performance, and models that can recombine patterns they have already learned — as humans do easily — to generalize to new situations more rapidly. An explicit goal of this project is to broaden research into why deep learning works. To this end, the investigators will integrate the research into education and establish a mentorship program for high school students from groups underrepresented in science.The goal of the research is to understand the role of representation theory in enabling efficient optimization and improved generalization of deep learning even in domains with approximate or unknown symmetry. This project pursues three lines of research that will broaden the impact of representation theory in deep learning beyond strict inductive biases. The first is the trade-off between the degree of symmetry in the model and the degree of symmetry in the domain. This line of research will study networks that combine equivariant and non-equivariant features. The second line of research will examine learning symmetry directly from data to improve generalization in domains without known symmetries. The third aim is to develop a theoretical basis for deep learning using quiver representations. This perspective reveals the symmetry of the structure of deep-learning models themselves, through their parameter spaces, even when the domains have no obvious symmetry.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.