CRII: RI: Principled Methods for Learning and Understanding of Neural Networks

深度神经网络在机器学习方面取得了突破性的成功，在面部识别、物体识别、异常检测和大规模监控辅助等各种任务上都取得了令人印象深刻的准确性。然而，深度神经网络在理论上并不能保证总是表现良好，而且它们可能会在以前看不见的数据或数据的微小/不可感知（对抗性）变化的情况下失败，尽管这种情况很少。例如，使用深度神经网络面部识别算法的家庭安全系统可能会将戴着像素化太阳镜的陌生人误认为房主;环境的轻微变化，例如下雨天，可能会导致基于计算机视觉的自动驾驶车辆将“停止”标志错误地识别为户外商业标志。在今天广泛使用的机器学习系统中存在这样的失败案例可能会使我们的日常生活甚至国家安全处于危险之中。使机器学习系统对这些失败案例具有鲁棒性的一种方法是设计保证提供最佳解决方案的算法，推广到看不见的场景，并对对抗性变化具有鲁棒性，即使攻击者完全了解算法。通过这项研究开发的方法将为解释“黑箱”深度神经网络提供理论基础，并在应用于高风险问题时为其性能提供保证。该项目将与研究生和本科教育相结合，促进计算机科学，应用数学，物理和商业研究人员之间的合作。通过该项目开发的软件程序将作为开源工具包发布，允许广泛传播给一系列领域的研究人员和从业人员。该项目将倡导通过学习理论、非凸优化和使用谱方法的一致性潜变量模型学习等技术，在理论上保证训练和理解神经网络。研究人员的目标是设计压缩的神经网络，理论上保证能够很好地泛化，适合具有内存限制的物联网设备，并且对对抗性示例具有鲁棒性。具体而言，该项目的技术目标分为三个重点。 (1)保证深度网络的训练。研究人员提出了一个理论上的理由，说明为什么深度残差网络比非残差网络更容易优化，因为每一层在预测标签时都提供了比弱基线更好的预言。研究人员计划使用两种方法来保证存在性并实现优于弱基线的预言：（a）利用理论上有保证的浅层卷积神经网络训练，也就是卷积字典学习，使用谱方法，以及（B）确保使用同伦变换来“锐化”网络的目标景观的局部最优值以逃离局部最优值，因为最近在逃离局部最优值方面的进展表明SGD不会卡在具有小直径的尖锐局部最优值处。(2)分析压缩深度神经网络的泛化能力。研究人员将使用张量化张量分解引入深度神经网络压缩，并为泛化误差制定更严格的界限，其中考虑了网络的输入分布和可压缩性。 (3)可靠的深度神经网络对最坏的攻击者具有鲁棒性。为了提供一个通用的防御机制，针对最坏的可能的对抗性的例子，使用极大极小公式，研究者建议分析非线性单-该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响力进行评估，被认为值得支持审查标准。

Deep neural networks have elicited breakthrough successes in machine learning by achieving impressive accuracies on diverse tasks such as facial recognition, object identification, anomaly detection and monitoring assistance on a large scale. However, deep neural networks are not theoretically guaranteed to always perform well, and they could, although rarely, fail in the presence of previously unseen data or small/imperceptible (adversarial) changes to the data. For instance, a home security system using a deep neural network facial recognition algorithm could mistake a stranger wearing pixelated sunglasses for the homeowner; a slight change of the environment, such as a rainy day, could cause a computer vision based autonomous driving vehicle to wrongly recognize a "STOP" sign as an outdoor commercial sign. The existence of such failure cases in widely used machine learning systems today could put our daily lives and even national security at risk. One way to make machine learning systems robust against these failure cases is to design algorithms that are guaranteed to provide an optimal solution, generalize to unseen scenarios and be robust to adversarial changes even if the attacker is given full knowledge of the algorithm. The methods developed via this research will provide theoretical bases that explain "black-box" deep neural networks and provide guarantees over their performance when applied to high-stakes problems. The project will be integrated with graduate and undergraduate education, fostering collaboration between researchers from Computer Science, Applied Math, Physics and Business. Software programs developed via this project will be released as an open-source toolkit, allowing widespread dissemination to researchers and practitioners in a range of fields.This project will advocate theoretically guaranteed training and understanding of neural networks via techniques from learning theory, nonconvex optimization and consistent latent variable model learning using spectral methods. The investigator's goal is to design compressed neural networks that are theoretically guaranteed to generalize well, fit into Internet of Things devices with memory constraints, and are robust to adversarial examples. Concretely, the technical aims of the project are divided into three thrusts. (1) Guaranteed training of deep nets. The investigator proposes to develop a theoretical justification of why deep residual networks are easier to optimize than non-residual ones when each layer provides a better-than-a-weak-baseline oracle in predicting labels. The investigator plans to use two approaches to guarantee existence and implement the better-than-a-weak-baseline oracle: (a) exploiting theoretically guaranteed training of shallow convolutional neural networks, a.k.a. convolutional dictionary learning, using spectral methods and (b) ensuring escaping from local optima using Homotopy transformations to "sharpen" local optima of network's objective landscape as recent advances in escaping from local optima showed that SGD will not get stuck at sharp local optima with small diameters. (2) Analyzing generalization ability of compressed deep neural networks. The investigator will introduce deep neural network compression using tensorized tensor decomposition, and develop tighter bounds for generalization error, which takes the input distribution and the compressibility of the network into account. (3) Reliable deep neural networks robust to the worst attackers. To provide a universal defense mechanism against the worst possible adversarial examples using a minimax formulation, the investigator proposes to analyze the robustness of nonlinear single-layer neural nets using tensor decomposition method and ultimately design universal defense mechanisms for deep neural nets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.