Mathematical Principles for Neural Network Design

尽管神经网络已经显示出解决复杂问题的能力，但人们对它们的行为仍然知之甚少。神经网络现在用于许多目的，从自动驾驶到医学图像分析。在每种情况下，都需要一组不同的属性，例如对新数据的良好泛化或对噪声的鲁棒性。目前，还没有一种原则性的方法来设计神经网络，使其具有手头任务所需的特定特征集。这种缺乏理论基础的情况意味着，成功不是从严格的理解中设计出来的最佳算法，而是渐进式的，来自试错实验，而失败可能是灾难性的，令人惊讶。该研究计划的长期目标是获得神经网络中设计选择如何影响其性能的正式数学理解，并使用这些理论结果为从业者提供可操作的见解。我们的研究有以下短期目标。目标1：量化归纳偏差神经网络的结构及其优化过程决定了网络可以表达和学习的函数集，本质上是对某些函数的归纳偏差进行编码，远离其他函数。在这个目标中，我们将得出关于神经网络的设计如何影响这种归纳偏差的理论结果。目标二：将算法与数据匹配这个目标将考虑如何利用不同神经网络的归纳偏差来改进深度学习算法。我们将推导出用于识别解决特定任务所需的归纳偏差的方法，并将设计学习方法，以高度控制神经网络将学习哪些函数。目标3：提高安全性我们将使用我们对归纳偏差的数学理解来提高深度学习算法的安全性。我们将展示什么时候可以从神经网络计算的函数中提取有关神经网络参数的信息，以及如何防止这种情况。我们的工作将保护神经网络和用于训练它们的数据的隐私，并将防止对抗性攻击。总的来说，这项研究将为神经网络的原则性设计提供急需的工具，从而显著提高算法的性能和可靠性，这些算法在整个社会中越来越重要。这将对从机器人到能源的各个领域产生影响。了解深度学习创新背后的数学原理也将有助于保持加拿大在人工智能领域的领先地位。我们的工作将培养HQP成为深度学习理论和工程交叉领域的领先创新者，这是学术界和工业界都非常需要的技能的协同组合。

Even as neural networks have shown their power in solving complex problems, their behavior remains poorly understood. Neural networks are now used for many purposes, from autonomous driving to medical image analysis. In each of these situations, a different set of properties is needed, such as good generalization to novel data or robustness to noise. Currently, there is no principled way to design neural networks to possess the particular set of characteristics needed for the task at hand. This lack of theoretical grounding means that instead of optimal algorithms designed from rigorous understanding, successes are incremental and arise from trial-and-error experimentation, while failures can be catastrophic and come as a surprise. The long-term goal of this research program is to gain a formal mathematical understanding of how design choices in neural networks affect their performance, and to use these theoretical results to derive actionable insights for practitioners. Our research has the following short-term objectives. Objective 1: Quantifying inductive biases The structure of a neural network and its optimization process determine the sets of functions that the network can express and learn, essentially encoding an inductive bias towards certain functions and away from others. In this objective, we will derive theoretical results on how the design of a neural network influences this inductive bias. Objective 2: Matching algorithms to data This objective will consider how the inductive biases of different neural networks can be leveraged to improve deep learning algorithms. We will derive methods for identifying the inductive biases that are needed to solve a particular task, and will design learning methods that give a high degree of control over which functions a neural network will learn. Objective 3: Improving security We will use our mathematical understanding of inductive biases to improve the security of deep learning algorithms. We will show when it is possible to extract information about the parameters of a neural network from the function it computes, as well as how to guard against this. Our work will protect the privacy of neural networks and the data used to train them, and will prevent adversarial attacks. Overall, this research will provide much-needed tools for the principled design of neural networks, allowing for significant increases in performance and reliability from algorithms that are increasingly essential across society. This will have an impact on fields from robotics to energy. Understanding the mathematical principles behind deep learning innovation will also help maintain Canada's preeminent position in AI. Our work will train HQP to be leading innovators in the intersection of deep learning theory and engineering, a synergistic combination of skills that is much in demand within both academia and industry.