Physics Based Architectures for Deep Neural Networks

基于物理的深度神经网络架构

• 批准号: RGPIN-2019-04052
• 负责人: Haber, Eldad
• 金额: $ 2.7万
• 依托单位: University of British Columbia
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=737368
• 关键词: Physics; Based; Architectures; Deep; Neural

Deep Neural Networks (DNNs) have been the engine behind the recent deep learning revolution. Such networks have demonstrated remarkable capabilities in solving image classification problems, image segmentation problems and other computer vision tasks. Such tasks are common in applications such as face recognition, autonomous cars and medical image understanding. In recent years, modifications to these networks have been proposed, and in particular, residual networks have shown to improve over existing techniques, and yield even better results. More specifically, residual networks are easier to train, have been used with hundreds and even thousands of layers and can be made memory efficient.  Encouraged from these results, a few researchers have attempted to explain the success of such networks by analyzing them as discrete dynamical systems, using theory from ordinary differential equations and optimal control. While initial results are interesting, there is still a large gap between the current technology of training and understanding DNNs, to other fields that use similar methodologies, such as time dependent optimal control, inverse problems and partial differential equations based optimization. Furthermore,  DNNs do not have an over-arching continuous analog which makes them somewhat add-hoc and problem/data dependent. Understanding how to move networks between scales, changing the network parameters and understanding its properties are all difficult and in many cases, impossible. The goal of this work is to suggest and develop new ideas for the understanding of DNNs. In particular, we exploit the interpretation of neural networks as dynamical systems in order to propose physics based, or physically motivated networks. These networks represent physical processes such as nonlinear diffusion and nonlinear wave propagation similar to the partial differential equations that have been used for compressed sensing. Such networks can have favorable properties in term of their overall character and may be more amendable to fast optimization algorithms that utilize this structure such as multigrid methods. I will be training 3 Ph.D. students as well as 3-5 undergraduate research assistants in the fundamentals of machine learning and its application to problems in geoscience imaging.Overall, this research program will increase scientific knowledge in the computing of machine learning and will contribute to the growing number of companies working on AI in Canada.  We expect students trained via the proposed research will pursue promising professional careers in either academia or industry.

深度神经网络（DNN）一直是最近深度学习革命背后的引擎。这种网络在解决图像分类问题、图像分割问题和其他计算机视觉任务方面表现出了卓越的能力。这些任务在人脸识别、自动汽车和医学图像理解等应用中很常见。近年来，已经提出了对这些网络的修改，特别是残差网络已经显示出对现有技术的改进，并且产生甚至更好的结果。更具体地说，残差网络更容易训练，已经被用于数百甚至数千层，并且可以提高内存效率。受这些结果的鼓舞，一些研究人员试图通过将其分析为离散动力系统来解释这种网络的成功，使用常微分方程和最优控制理论。 虽然最初的结果很有趣，但目前的训练和理解DNN的技术与使用类似方法的其他领域之间仍然存在很大的差距，例如时间相关的最优控制，反问题和基于偏微分方程的优化。此外，DNN不具有过度连续的模拟，这使得它们在某种程度上具有附加性和问题/数据依赖性。了解如何在尺度之间移动网络，改变网络参数并了解其属性都很困难，在许多情况下是不可能的。这项工作的目标是为理解DNN提出和发展新的想法。特别是，我们利用神经网络作为动力系统的解释，以提出基于物理或物理动机的网络。这些网络表示物理过程，例如非线性扩散和非线性波传播，类似于已用于压缩传感的偏微分方程。这样的网络可以在其整体特征方面具有有利的性质，并且可以更易于使用利用这种结构的快速优化算法，例如多重网格方法。 我将培养3名博士。学生以及3-5名本科生研究助理，学习机器学习的基础知识及其在地球科学成像问题中的应用。总体而言，这项研究计划将增加机器学习计算方面的科学知识，并将为加拿大越来越多的人工智能公司做出贡献。我们希望通过拟议的研究培训的学生将在学术界或行业