Evaluating and Improving Deep Neural Networks

深度神经网络最近推动了图像理解、语言理解、基因组学、计算化学、游戏和机器人等领域的最新应用。特别是深度生成模型在过去几年中取得了快速进展，网络能够产生可信的图像或语音信号。尽管这些应用领域各不相同，但研究人员和实践者面临的挑战和挫折有很多共同点。在昂贵的图形处理单元（GPU）硬件上训练神经网络可能需要数周时间。与上一代机器学习算法相比，该算法有更多需要调整的旋钮。在深度生成模型的情况下，很难确定网络是在学习对数据建模，还是只是记忆它们的训练样本。******这些问题的解决方案将在工业和科学研究的一系列应用领域产生巨大影响。我建议使用结构化概率建模技术来驯服神经网络的复杂性。在接下来的五年里，我将专注于两个主线：改进神经网络的优化，以及评估深度生成模型。******即使使用现代GPU硬件，训练神经网络也需要数周时间。之前，我介绍了一种从成本函数曲率的概率模型中推导有效二阶优化算法的方法。这使得训练许多类型的神经网络的速度大大提高。我计划将该技术扩展到用于图像、视频和文本理解的最先进的体系结构中，并将该技术扩展到在集群而不是单个处理器上进行训练。最终的结果将是一个通用的神经网络训练算法，它是有效的和可扩展的，并且需要很少的手工调整。******评估生成模型的主要障碍是难以计算分配给一个配置的概率，以及难以确定一个人的估计有多准确。（发表在这一主题上的论文往往会警告称，报告的概率可能极其不准确。）评价困难被认为是阻碍生成建模科学发展的主要因素之一。我的目标是开发获得概率的置信区间的技术，这些置信区间既紧密又准确，这样我们就可以对生成模型的评估有信心。这将使生成模型的严格实证研究成为可能，从而使我们能够改进它们********

Deep neural networks have recently pushed forward the state-of-the-art in applications as diverse as image understanding, language understanding, genomics, computational chemistry, game playing, and robotics. Deep generative models in particular have seen rapid progress in the past few years, with networks able to produce plausible images or speech signals. As diverse as these application areas are, the challenges and frustrations facing the researchers and practitioners have much in common. Neural networks can take weeks to train on expensive Graphics Processing Unit (GPU) hardware. The algorithms have many more knobs which need to be tweaked, compared with the previous generation of machine learning algorithms. In the case of deep generative models, it can be hard to determine if the networks are learning to model the data or simply memorizing their training examples.******Solutions to any of these issues would have enormous impact across a range of application areas, both in industry and in scientific research. I propose to tame the complexity of neural networks using the techniques of structured probabilistic modeling. Over the next five years, I will focus on two main threads: improving the optimization of neural networks, and evaluating deep generative models. ******Training neural networks can take weeks, even with modern GPU hardware. Previously, I introduced a method for deriving efficient second-order optimization algorithms from probabilistic models of the curvature of a cost function. This led to large speedups in training many types of neural nets. I plan to extend this technique to state-of-the-art architectures for image, video, and text understanding, and to scale up the technique to training on a cluster rather than an individual processor. The end result will be a general-purpose neural net training algorithm which is efficient and scalable and requires little hand-tweaking.******The main obstacle to evaluating generative models is the intractability of computing the probability assigned to a configuration, coupled with the difficulty of determining how accurate one's estimates are. (Papers published on the topic tend to include caveats that the reported probabilities may be extremely inaccurate.) The difficulty of evaluation is considered one of the main factors holding back scientific progress on generative modeling. I aim to develop techniques for obtaining confidence intervals for the probabilities which are both tight and accurate, so that we can have confidence in our evaluation of generative models. This will enable rigorous empirical study of generative models, which in turn will allow us to improve them.********