RI: Small: Understanding the Inductive Bias Caused by Invariance and Multi Scale in Neural Networks

深度神经网络最近对世界产生了巨大的影响。它们被广泛用于理解语音、翻译语言和分析图像的系统中。尽管它们产生了巨大的影响，但研究人员仍然缺乏对这些网络的许多基本属性的严格理解。因此，新网络的设计在很大程度上是通过反复试验进行的。尽管总体上非常有效，但这些系统有时会被看似非常简单的示例所愚弄，这些示例与其他易于处理的示例类似。本研究的目的是从理论上更好地理解一类重要的神经网络，即卷积神经网络，它被广泛应用于理解图像和音频信号。该项目的重点是了解哪些问题对于CNN来说是容易的还是困难的。这种理解可以帮助我们预测这些网络中的偏差，并了解网络的设计将如何影响其行为。该项目将为研究生、本科生和高中生提供研究机会，特别是接触到代表不足的群体的学生。CNN与许多其他机器学习方法不同的两个关键特性是它们能够自然地结合多尺度分析和不变性或等价性。这一性质使得能够有效地处理在网格数据上采样的图像和信号的移位不变网络的构造，以及最近处理集合和图的网络的构造，结合了集合置换和图同构的等变运算。多尺度表示通过它们的深度在这些网络中自然地出现。本研究的重点在于更好地理解多尺度、不变性和等方差在神经网络中的作用。它将研究移位不变性和多尺度表示如何影响神经网络训练的动力学。我们的方法将建立在最近的结果基础上，这些结果表明，大规模过参数神经网络可以表示为核方法。分析这些核的性质将有助于我们理解网络体系结构与其归纳偏差之间的关系。所揭示的见解有可能提供一种更有原则的方式来控制这些偏见。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks have had a huge recent impact on the world. They are widely used in systems that understand speech, translate language, and analyze images. In spite of their great impact, researchers still lack a rigorous understanding of many of the basic properties of these networks. As a consequence, new networks are largely designed laboriously, through trial and error. And although extremely effective overall, these systems are sometimes fooled by examples that seem very simple, and similar to other examples that are easily handled. This research aims to provide a better theoretical understanding of an important class of neural networks, called Convolutional Neural Networks (CNNs), which are widely used in understanding images and audio signals. The project focuses on understanding what problems will be easy or difficult for CNNs. This understanding can help us to predict biases in these networks and understand how the design of a network will affect its behavior. The project will provide research opportunities for graduate, undergraduate and high school students, particularly reaching out to students from underrepresented groups.Two key properties that distinguish CNNs from many other approaches to machine learning are their ability to naturally incorporate multiscale analysis and invariance or equivariance. This property has enabled the construction of shift invariant networks that effectively deal with images and signals sampled on grid data, and more recently of networks that handle sets and graphs, incorporating operations that are equivariant to set permutation and graph isomorphism. Multiscale representations naturally arise in these networks through their depth. This research focuses on gaining a better understanding of the role of multiscale, invariance and equivariance in neural networks. It will study how shift invariance and multiscale representations affect the dynamics of neural network training. Our approach will build on recent results showing that massively overparameterized neural networks can be represented as kernel methods. Analyzing the properties of these kernels will help us understand the relationship between a network's architecture and its inductive biases. The insights revealed have the potential to provide a more principled way to control these biases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.