Evaluating and Improving Deep Neural Networks

深度神经网络最近在图像理解、语言理解、基因组学、计算化学、游戏和机器人等各种应用中推动了最先进的应用。特别是深度生成模型在过去几年中取得了快速进展，网络能够产生可信的图像或语音信号。尽管这些应用领域千差万别，但研究人员和实践者面临的挑战和挫折有许多共同之处。神经网络可能需要数周时间来训练昂贵的图形处理单元(GPU)硬件。与上一代机器学习算法相比，这些算法有更多需要调整的旋钮。在深度生成性模型的情况下，可能很难确定网络是在学习对数据进行建模，还是只是记住它们的训练示例。*这些问题的解决方案将在工业和科学研究的一系列应用领域产生巨大影响。我建议使用结构化概率建模技术来驯服神经网络的复杂性。在接下来的五年里，我将把重点放在两个主线上：改进神经网络的优化，以及评估深度生成模型。*训练神经网络可能需要数周时间，即使使用现代GPU硬件也是如此。在此之前，我介绍了一种从成本函数曲率的概率模型导出高效二阶优化算法的方法。这导致了在训练多种类型的神经网络时的大幅加速。我计划将这项技术扩展到用于图像、视频和文本理解的最先进的体系结构，并将该技术扩展到在集群而不是单个处理器上进行培训。最终结果将是一种通用的神经网络训练算法，它高效、可伸缩，几乎不需要手动调整。*评估生成模型的主要障碍是难以计算分配给某个配置的概率，以及难以确定一个人的估计有多准确。(发表在这一主题上的论文往往包括警告，即报道的概率可能极不准确。)评价的困难被认为是阻碍产生式建模科学发展的主要因素之一。我的目标是开发获得概率的可信区间的技术，这些概率既紧凑又准确，这样我们就可以对生成模型的评估有信心。这将使我们能够对生成模型进行严格的实证研究，从而使我们能够改进它们。

Deep neural networks have recently pushed forward the state-of-the-art in applications as diverse as image understanding, language understanding, genomics, computational chemistry, game playing, and robotics. Deep generative models in particular have seen rapid progress in the past few years, with networks able to produce plausible images or speech signals. As diverse as these application areas are, the challenges and frustrations facing the researchers and practitioners have much in common. Neural networks can take weeks to train on expensive Graphics Processing Unit (GPU) hardware. The algorithms have many more knobs which need to be tweaked, compared with the previous generation of machine learning algorithms. In the case of deep generative models, it can be hard to determine if the networks are learning to model the data or simply memorizing their training examples.******Solutions to any of these issues would have enormous impact across a range of application areas, both in industry and in scientific research. I propose to tame the complexity of neural networks using the techniques of structured probabilistic modeling. Over the next five years, I will focus on two main threads: improving the optimization of neural networks, and evaluating deep generative models. ******Training neural networks can take weeks, even with modern GPU hardware. Previously, I introduced a method for deriving efficient second-order optimization algorithms from probabilistic models of the curvature of a cost function. This led to large speedups in training many types of neural nets. I plan to extend this technique to state-of-the-art architectures for image, video, and text understanding, and to scale up the technique to training on a cluster rather than an individual processor. The end result will be a general-purpose neural net training algorithm which is efficient and scalable and requires little hand-tweaking.******The main obstacle to evaluating generative models is the intractability of computing the probability assigned to a configuration, coupled with the difficulty of determining how accurate one's estimates are. (Papers published on the topic tend to include caveats that the reported probabilities may be extremely inaccurate.) The difficulty of evaluation is considered one of the main factors holding back scientific progress on generative modeling. I aim to develop techniques for obtaining confidence intervals for the probabilities which are both tight and accurate, so that we can have confidence in our evaluation of generative models. This will enable rigorous empirical study of generative models, which in turn will allow us to improve them.********