Mathematical Principles for Neural Network Design

点击翻译按钮获取中文摘要

Even as neural networks have shown their power in solving complex problems, their behavior remains poorly understood. Neural networks are now used for many purposes, from autonomous driving to medical image analysis. In each of these situations, a different set of properties is needed, such as good generalization to novel data or robustness to noise. Currently, there is no principled way to design neural networks to possess the particular set of characteristics needed for the task at hand. This lack of theoretical grounding means that instead of optimal algorithms designed from rigorous understanding, successes are incremental and arise from trial-and-error experimentation, while failures can be catastrophic and come as a surprise. The long-term goal of this research program is to gain a formal mathematical understanding of how design choices in neural networks affect their performance, and to use these theoretical results to derive actionable insights for practitioners. Our research has the following short-term objectives. Objective 1: Quantifying inductive biases The structure of a neural network and its optimization process determine the sets of functions that the network can express and learn, essentially encoding an inductive bias towards certain functions and away from others. In this objective, we will derive theoretical results on how the design of a neural network influences this inductive bias. Objective 2: Matching algorithms to data This objective will consider how the inductive biases of different neural networks can be leveraged to improve deep learning algorithms. We will derive methods for identifying the inductive biases that are needed to solve a particular task, and will design learning methods that give a high degree of control over which functions a neural network will learn. Objective 3: Improving security We will use our mathematical understanding of inductive biases to improve the security of deep learning algorithms. We will show when it is possible to extract information about the parameters of a neural network from the function it computes, as well as how to guard against this. Our work will protect the privacy of neural networks and the data used to train them, and will prevent adversarial attacks. Overall, this research will provide much-needed tools for the principled design of neural networks, allowing for significant increases in performance and reliability from algorithms that are increasingly essential across society. This will have an impact on fields from robotics to energy. Understanding the mathematical principles behind deep learning innovation will also help maintain Canada's preeminent position in AI. Our work will train HQP to be leading innovators in the intersection of deep learning theory and engineering, a synergistic combination of skills that is much in demand within both academia and industry.