CRII: RI: Principled Methods for Learning and Understanding of Neural Networks

CRII：RI：学习和理解神经网络的原则方法

• 批准号: 1850220
• 负责人: Furong Huang
• 金额: $ 17.5万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-05-01 至 2022-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1850220&HistoricalAwards=false
• 关键词: CRII; RI; Principled; Methods; Learning

Deep neural networks have elicited breakthrough successes in machine learning by achieving impressive accuracies on diverse tasks such as facial recognition, object identification, anomaly detection and monitoring assistance on a large scale. However, deep neural networks are not theoretically guaranteed to always perform well, and they could, although rarely, fail in the presence of previously unseen data or small/imperceptible (adversarial) changes to the data. For instance, a home security system using a deep neural network facial recognition algorithm could mistake a stranger wearing pixelated sunglasses for the homeowner; a slight change of the environment, such as a rainy day, could cause a computer vision based autonomous driving vehicle to wrongly recognize a "STOP" sign as an outdoor commercial sign. The existence of such failure cases in widely used machine learning systems today could put our daily lives and even national security at risk. One way to make machine learning systems robust against these failure cases is to design algorithms that are guaranteed to provide an optimal solution, generalize to unseen scenarios and be robust to adversarial changes even if the attacker is given full knowledge of the algorithm. The methods developed via this research will provide theoretical bases that explain "black-box" deep neural networks and provide guarantees over their performance when applied to high-stakes problems. The project will be integrated with graduate and undergraduate education, fostering collaboration between researchers from Computer Science, Applied Math, Physics and Business. Software programs developed via this project will be released as an open-source toolkit, allowing widespread dissemination to researchers and practitioners in a range of fields.This project will advocate theoretically guaranteed training and understanding of neural networks via techniques from learning theory, nonconvex optimization and consistent latent variable model learning using spectral methods. The investigator's goal is to design compressed neural networks that are theoretically guaranteed to generalize well, fit into Internet of Things devices with memory constraints, and are robust to adversarial examples. Concretely, the technical aims of the project are divided into three thrusts. (1) Guaranteed training of deep nets. The investigator proposes to develop a theoretical justification of why deep residual networks are easier to optimize than non-residual ones when each layer provides a better-than-a-weak-baseline oracle in predicting labels. The investigator plans to use two approaches to guarantee existence and implement the better-than-a-weak-baseline oracle: (a) exploiting theoretically guaranteed training of shallow convolutional neural networks, a.k.a. convolutional dictionary learning, using spectral methods and (b) ensuring escaping from local optima using Homotopy transformations to "sharpen" local optima of network's objective landscape as recent advances in escaping from local optima showed that SGD will not get stuck at sharp local optima with small diameters. (2) Analyzing generalization ability of compressed deep neural networks. The investigator will introduce deep neural network compression using tensorized tensor decomposition, and develop tighter bounds for generalization error, which takes the input distribution and the compressibility of the network into account. (3) Reliable deep neural networks robust to the worst attackers. To provide a universal defense mechanism against the worst possible adversarial examples using a minimax formulation, the investigator proposes to analyze the robustness of nonlinear single-layer neural nets using tensor decomposition method and ultimately design universal defense mechanisms for deep neural nets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络在机器学习方面取得了突破性的成功，在大规模的面部识别、目标识别、异常检测和监控辅助等各种任务中取得了令人印象深刻的准确性。然而，深度神经网络在理论上并不能保证总是表现良好，而且它们可能会在存在以前未见的数据或数据的微小/不可察觉的(对抗性)变化时失败，尽管这种情况很少发生。例如，使用深度神经网络面部识别算法的家庭安全系统可能会将戴着像素化太阳镜的陌生人误认为房主；环境的微小变化，如下雨天，可能会导致基于计算机视觉的自动驾驶车辆将“停车”标志错误地识别为户外商业标志。如今广泛使用的机器学习系统中存在这样的故障案例，可能会危及我们的日常生活，甚至国家安全。使机器学习系统对这些故障情况具有健壮性的一种方法是设计算法，即使攻击者完全了解算法，也可以保证提供最优解决方案，对未知场景进行推广，并对对抗性变化具有健壮性。通过这项研究开发的方法将为解释“黑箱”深度神经网络提供理论基础，并为其应用于高风险问题时的性能提供保证。该项目将与研究生和本科教育相结合，促进来自计算机科学、应用数学、物理和商业的研究人员之间的合作。通过这个项目开发的软件程序将作为一个开源工具包发布，允许在一系列领域的研究人员和从业者广泛传播。该项目将倡导通过学习理论、非凸优化和使用谱方法进行一致的潜变量模型学习的技术来进行神经网络的理论保证培训和理解。研究人员的目标是设计压缩的神经网络，从理论上保证具有良好的泛化能力，适合具有存储限制的物联网设备，并且对敌意例子具有健壮性。具体地说，该项目的技术目标分为三个方面。(1)保证深度网络的训练。这位研究人员建议，当每一层在预测标签方面提供一个比弱基线更好的预言时，为什么深层残差网络比非残差网络更容易优化，这一点可以从理论上得到证明。研究人员计划使用两种方法来保证存在并实现比弱基线更好的先知：(A)利用理论上有保证的浅卷积神经网络(又名)的训练。卷积字典学习，使用谱方法和(B)使用同伦变换确保摆脱局部最优，以“锐化”网络目标景观的局部最优，因为最近在摆脱局部最优方面的进展表明，SGD不会陷入小直径的尖锐局部最优。(2)分析了压缩深度神经网络的泛化能力。研究人员将引入张量张量分解的深度神经网络压缩，并在考虑输入分布和网络的可压缩性的情况下建立更严格的泛化误差界。(3)对最坏的攻击者具有健壮性的可靠的深度神经网络。为了利用极小极大公式提供一种针对最糟糕的可能的敌对例子的通用防御机制，研究人员建议使用张量分解方法来分析非线性单层神经网络的健壮性，并最终设计用于深层神经网络的通用防御机制。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Understanding Generalization through Visualizations

• 发表时间: 2019-06
• 期刊: ArXiv
• 作者: W. R. Huang;Z. Emam;Micah Goldblum;Liam H. Fowl;J. K. Terry;Furong Huang;T. Goldstein

An end-to-end Differentially Private Latent Dirichlet Allocation Using a Spectral Algorithm

• 发表时间: 2018-05
• 作者: Seyed-Alireza Esmaeili;Furong Huang

Adaptive Learning Rates with Maximum Variation Averaging

• 发表时间: 2020-06
• 期刊: ArXiv
• 作者: Chen Zhu;Yu Cheng;Zhe Gan;Furong Huang;Jingjing Liu;T. Goldstein

Fast GPU Convolution for CP-Decomposed Tensorial Neural Networks

• DOI: 10.1007/978-3-030-55180-3_35
• 发表时间: 2020-09
• 作者: A. Reustle;Tahseen Rabbani;Furong Huang

Convolutional Tensor-Train LSTM for Spatio-temporal Learning

• 发表时间: 2020-02
• 期刊: ArXiv
• 作者: Jiahao Su;Wonmin Byeon;Furong Huang;J. Kautz;Anima Anandkumar