Mathematical Principles for Neural Network Design

神经网络设计的数学原理

• 批准号: RGPIN-2021-03864
• 负责人: Rolnick, David
• 金额: $ 2.11万
• 依托单位: McGill University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=750380
• 关键词: Mathematical; Principles; Neural; Network; Design

Even as neural networks have shown their power in solving complex problems, their behavior remains poorly understood. Neural networks are now used for many purposes, from autonomous driving to medical image analysis. In each of these situations, a different set of properties is needed, such as good generalization to novel data or robustness to noise. Currently, there is no principled way to design neural networks to possess the particular set of characteristics needed for the task at hand. This lack of theoretical grounding means that instead of optimal algorithms designed from rigorous understanding, successes are incremental and arise from trial-and-error experimentation, while failures can be catastrophic and come as a surprise. The long-term goal of this research program is to gain a formal mathematical understanding of how design choices in neural networks affect their performance, and to use these theoretical results to derive actionable insights for practitioners. Our research has the following short-term objectives. Objective 1: Quantifying inductive biases The structure of a neural network and its optimization process determine the sets of functions that the network can express and learn, essentially encoding an inductive bias towards certain functions and away from others. In this objective, we will derive theoretical results on how the design of a neural network influences this inductive bias. Objective 2: Matching algorithms to data This objective will consider how the inductive biases of different neural networks can be leveraged to improve deep learning algorithms. We will derive methods for identifying the inductive biases that are needed to solve a particular task, and will design learning methods that give a high degree of control over which functions a neural network will learn. Objective 3: Improving security We will use our mathematical understanding of inductive biases to improve the security of deep learning algorithms. We will show when it is possible to extract information about the parameters of a neural network from the function it computes, as well as how to guard against this. Our work will protect the privacy of neural networks and the data used to train them, and will prevent adversarial attacks. Overall, this research will provide much-needed tools for the principled design of neural networks, allowing for significant increases in performance and reliability from algorithms that are increasingly essential across society. This will have an impact on fields from robotics to energy. Understanding the mathematical principles behind deep learning innovation will also help maintain Canada's preeminent position in AI. Our work will train HQP to be leading innovators in the intersection of deep learning theory and engineering, a synergistic combination of skills that is much in demand within both academia and industry.

尽管神经网络已经显示出其解决复杂问题的能力，但人们对它们的行为仍然知之甚少。神经网络现在用于多种用途，从自动驾驶到医学图像分析。在每种情况下，都需要一组不同的属性，例如对新数据的良好泛化或对噪声的鲁棒性。目前，还没有原则性的方法来设计神经网络来拥有手头任务所需的一组特定特征。缺乏理论基础意味着，不是从严格的理解中设计出最佳算法，成功是渐进的，来自反复试验，而失败可能是灾难性的，令人惊讶。该研究计划的长期目标是对神经网络中的设计选择如何影响其性能获得正式的数学理解，并利用这些理论结果为从业者得出可行的见解。我们的研究有以下短期目标。目标 1：量化归纳偏差 神经网络的结构及其优化过程决定了网络可以表达和学习的函数集，本质上编码了针对某些函数而远离其他函数的归纳偏差。在此目标中，我们将得出关于神经网络的设计如何影响这种归纳偏差的理论结果。目标 2：将算法与数据匹配该目标将考虑如何利用不同神经网络的归纳偏差来改进深度学习算法。我们将推导识别解决特定任务所需的归纳偏差的方法，并将设计对神经网络将学习的功能进行高度控制的学习方法。目标 3：提高安全性我们将利用对归纳偏差的数学理解来提高深度学习算法的安全性。我们将展示何时可以从神经网络计算的函数中提取有关神经网络参数的信息，以及如何防范这种情况。我们的工作将保护神经网络的隐私以及用于训练神经网络的数据，并防止对抗性攻击。总体而言，这项研究将为神经网络的原理设计提供急需的工具，从而显着提高整个社会日益重要的算法的性能和可靠性。这将对从机器人到能源等领域产生影响。了解深度学习创新背后的数学原理也将有助于保持加拿大在人工智能领域的卓越地位。我们的工作将培训 HQP 成为深度学习理论与工程交叉领域的领先创新者，这是学术界和工业界都非常需要的技能的协同组合。