Collaborative Research: SCALE MoDL: Representation Theoretic Foundations of Deep Learning

在过去的十年里，深度学习在整个社会产生了革命性的影响。然而，进展往往依赖于启发式方法、海量数据和强大的计算能力。这带来了有限的理论理解，有时会导致泛化失败，在极端情况下表现脆弱。这个项目将通过为使用表征理论的深度学习发展坚实的理论基础来解决这些限制，表征理论是对对称性的数学研究。对称性在人类推理中起着关键作用。更好地理解对称性在深度学习中所起的作用，将解锁各种改进的模型。这些模型包括可以从科学知识而不仅仅是原始数据中学习的模型，具有可靠、有保证性能的模型，以及可以重新组合他们已经学习的模式的模型-就像人类很容易做到的那样-以更快地概括到新的情况。这个项目的一个明确目标是扩大对深度学习为什么有效的研究。为此，研究人员将把这项研究整合到教育中，并为科学中代表性不足的群体的高中生建立一个导师计划。研究的目标是了解表征理论在实现深度学习的高效优化和改进泛化方面的作用，即使在具有近似或未知对称性的领域也是如此。这个项目致力于三个方面的研究，这些研究将扩大表征理论在深度学习中的影响，超越严格的归纳偏见。第一个是在模型中的对称度和域中的对称度之间的权衡。这一系列研究将研究结合了等变和非等变特征的网络。第二条线的研究将检验直接从数据中学习对称性，以改进在没有已知对称性的领域中的泛化。第三个目标是为使用箭筒表征的深度学习奠定理论基础。这一观点揭示了深度学习模型本身结构的对称性，通过它们的参数空间，即使在领域没有明显的对称性的情况下也是如此。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In the past decade, deep learning has had transformative impacts across society. However, progress has often relied on heuristic methods, massive data, and great computing power. This comes with limited theoretical understanding and has at times given rise to failures of generalization and vulnerable performance in extreme scenarios. This project will address these limitations by developing strong theoretical foundations for deep learning using representation theory, which is the mathematical study of symmetry. Symmetry plays a key role in human reasoning. Greater understanding of the role symmetry plays in deep learning will unlock a variety of improved models. These include models that can learn from scientific knowledge and not just raw data, models with trustable, guaranteed performance, and models that can recombine patterns they have already learned — as humans do easily — to generalize to new situations more rapidly. An explicit goal of this project is to broaden research into why deep learning works. To this end, the investigators will integrate the research into education and establish a mentorship program for high school students from groups underrepresented in science.The goal of the research is to understand the role of representation theory in enabling efficient optimization and improved generalization of deep learning even in domains with approximate or unknown symmetry. This project pursues three lines of research that will broaden the impact of representation theory in deep learning beyond strict inductive biases. The first is the trade-off between the degree of symmetry in the model and the degree of symmetry in the domain. This line of research will study networks that combine equivariant and non-equivariant features. The second line of research will examine learning symmetry directly from data to improve generalization in domains without known symmetries. The third aim is to develop a theoretical basis for deep learning using quiver representations. This perspective reveals the symmetry of the structure of deep-learning models themselves, through their parameter spaces, even when the domains have no obvious symmetry.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.