RI: Small: Understanding the Inductive Bias Caused by Invariance and Multi Scale in Neural Networks

深度神经网络最近对世界产生了巨大的影响。它们广泛用于理解语音、翻译语言和分析图像的系统中。尽管它们的影响很大，但研究人员仍然缺乏对这些网络的许多基本属性的严格理解。因此，新的网络在很大程度上是通过试验和错误而费力地设计的。虽然总体上非常有效，但这些系统有时会被看似非常简单的例子所愚弄，并且与其他容易处理的例子相似。这项研究旨在更好地从理论上理解一类重要的神经网络，称为卷积神经网络（CNN），它广泛用于理解图像和音频信号。该项目的重点是了解哪些问题对CNN来说容易或困难。这种理解可以帮助我们预测这些网络中的偏差，并了解网络的设计如何影响其行为。该项目将为研究生，本科生和高中生提供研究机会，特别是接触来自代表性不足群体的学生。CNN与许多其他机器学习方法的两个关键特性是它们能够自然地结合多尺度分析和不变性或等变性。这一特性使得平移不变网络的建设，有效地处理图像和信号采样的网格数据，以及最近的网络，处理集和图形，将操作是等变的设置置换和图形同构。多尺度表示通过其深度自然地出现在这些网络中。这项研究的重点是获得更好的理解的多尺度，不变性和等变性在神经网络中的作用。它将研究移位不变性和多尺度表示如何影响神经网络训练的动态。我们的方法将建立在最近的结果表明，大规模过参数化的神经网络可以表示为内核方法。分析这些内核的属性将有助于我们理解网络结构与其归纳偏差之间的关系。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks have had a huge recent impact on the world. They are widely used in systems that understand speech, translate language, and analyze images. In spite of their great impact, researchers still lack a rigorous understanding of many of the basic properties of these networks. As a consequence, new networks are largely designed laboriously, through trial and error. And although extremely effective overall, these systems are sometimes fooled by examples that seem very simple, and similar to other examples that are easily handled. This research aims to provide a better theoretical understanding of an important class of neural networks, called Convolutional Neural Networks (CNNs), which are widely used in understanding images and audio signals. The project focuses on understanding what problems will be easy or difficult for CNNs. This understanding can help us to predict biases in these networks and understand how the design of a network will affect its behavior. The project will provide research opportunities for graduate, undergraduate and high school students, particularly reaching out to students from underrepresented groups.Two key properties that distinguish CNNs from many other approaches to machine learning are their ability to naturally incorporate multiscale analysis and invariance or equivariance. This property has enabled the construction of shift invariant networks that effectively deal with images and signals sampled on grid data, and more recently of networks that handle sets and graphs, incorporating operations that are equivariant to set permutation and graph isomorphism. Multiscale representations naturally arise in these networks through their depth. This research focuses on gaining a better understanding of the role of multiscale, invariance and equivariance in neural networks. It will study how shift invariance and multiscale representations affect the dynamics of neural network training. Our approach will build on recent results showing that massively overparameterized neural networks can be represented as kernel methods. Analyzing the properties of these kernels will help us understand the relationship between a network's architecture and its inductive biases. The insights revealed have the potential to provide a more principled way to control these biases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.