Evaluating and Improving Deep Neural Networks

评估和改进深度神经网络

• 批准号: RGPIN-2017-06050
• 负责人: Grosse, Roger
• 金额: $ 4.52万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=750039
• 关键词: Evaluating; Improving; Deep; Neural; Networks

Deep neural networks have recently pushed forward the state-of-the-art in applications as diverse as image understanding, language understanding, genomics, computational chemistry, game playing, and robotics. Deep generative models in particular have seen rapid progress in the past few years, with networks able to produce plausible images or speech signals. As diverse as these application areas are, the challenges and frustrations facing the researchers and practitioners have much in common. Neural networks can take weeks to train on expensive Graphics Processing Unit (GPU) hardware. The algorithms have many more knobs which need to be tweaked, compared with the previous generation of machine learning algorithms. In the case of deep generative models, it can be hard to determine if the networks are learning to model the data or simply memorizing their training examples.Solutions to any of these issues would have enormous impact across a range of application areas, both in industry and in scientific research. I propose to tame the complexity of neural networks using the techniques of structured probabilistic modeling. Over the next five years, I will focus on two main threads: improving the optimization of neural networks, and evaluating deep generative models. Training neural networks can take weeks, even with modern GPU hardware. Previously, I introduced a method for deriving efficient second-order optimization algorithms from probabilistic models of the curvature of a cost function. This led to large speedups in training many types of neural nets. I plan to extend this technique to state-of-the-art architectures for image, video, and text understanding, and to scale up the technique to training on a cluster rather than an individual processor. The end result will be a general-purpose neural net training algorithm which is efficient and scalable and requires little hand-tweaking.The main obstacle to evaluating generative models is the intractability of computing the probability assigned to a configuration, coupled with the difficulty of determining how accurate one's estimates are. (Papers published on the topic tend to include caveats that the reported probabilities may be extremely inaccurate.) The difficulty of evaluation is considered one of the main factors holding back scientific progress on generative modeling. I aim to develop techniques for obtaining confidence intervals for the probabilities which are both tight and accurate, so that we can have confidence in our evaluation of generative models. This will enable rigorous empirical study of generative models, which in turn will allow us to improve them.

深度神经网络最近推动了图像理解、语言理解、基因组学、计算化学、游戏和机器人等各种应用的发展。特别是深度生成模型在过去几年中取得了快速进展，网络能够产生合理的图像或语音信号。尽管这些应用领域多种多样，但研究人员和实践者面临的挑战和挫折有很多共同之处。神经网络可能需要数周的时间在昂贵的图形处理单元（GPU）硬件上进行训练。与上一代机器学习算法相比，这些算法有更多的旋钮需要调整。在深度生成模型的情况下，很难确定网络是在学习对数据建模，还是仅仅记住它们的训练示例。这些问题的解决方案将对工业和科学研究等一系列应用领域产生巨大影响。我建议使用结构化概率建模技术来驯服神经网络的复杂性。在接下来的五年里，我将专注于两个主要方面：改进神经网络的优化，以及评估深度生成模型。训练神经网络可能需要数周时间，即使使用现代GPU硬件也是如此。以前，我介绍了一种方法，用于从成本函数曲率的概率模型导出有效的二阶优化算法。这导致了训练许多类型神经网络的速度大幅提升。我计划将这项技术扩展到最先进的图像、视频和文本理解架构，并将这项技术扩展到集群而不是单个处理器上。最终的结果将是一个通用的神经网络训练算法，这是有效的，可扩展的，并需要很少的手工tweaking.The主要障碍，评估生成模型是计算分配给一个配置的概率的棘手性，再加上确定一个人的估计是多么准确的困难。（关于这个主题的论文往往包括警告，即报告的概率可能非常不准确。评估的难度被认为是阻碍生成式建模科学进步的主要因素之一。我的目标是开发技术来获得概率的置信区间，这些置信区间既紧密又准确，这样我们就可以对生成模型的评估充满信心。这将使我们能够对生成模型进行严格的实证研究，从而使我们能够改进它们。