RI: Small: Algorithms for accelerating optimization in deep learning

诸如图像或声音的复杂信号的智能处理通常由简单的非线性处理层的参数化嵌套层次来执行，如在深度神经网络、对象识别级联或语音前端中。各层参数的联合估计和最优体系结构的选择是一个困难的非凸优化问题，很难并行化，并且需要大量的人类专家的努力，这导致了实际中的次优系统。这项研究将开发一种新的、通用的数学策略来学习参数，在某种程度上，学习嵌套系统的结构，称为辅助坐标法(MAC)。MAC具有可证明的收敛特性，易于实现对单层的现有算法的重用，即使在参数导数不可用的情况下也适用，易于使用并行体系结构，并且通常在几次迭代内提供合理的模型。PI的研究和教学将向本科生介绍嵌套机器学习系统的设计，并为计算机科学研究生提供优化技能，以支持特定领域(机器学习、计算机视觉/语音等)。PI将通过发表优化、机器学习、计算机视觉和语音方面的论文来广泛传播研究结果。PI将使Matlab和C/C++代码可用于开发的算法，并将其用作他的优化和机器学习课程的教学辅助工具。PI将努力让不同群体的学生参与研究。MAC可以通过减少运行时间和人力，极大地促进目前在数据丰富的学科如机器学习、计算机视觉和语音，但也在工程和科学的其他领域开发的复杂模型的实际设计和评估。它还可以避免在树和其他分类器中构建手工制作的特征。它可能会给社会带来广泛和及时的好处，因为串行计算正在进入平台期，云计算正在成为一种商品，并且由于计算能力和数据可用性的增加，智能数据处理正在找到进入主流设备(手机、相机等)的途径。这项研究将沿着两个目标发展MAC。优化：PI将探索定义辅助坐标的不同方法；解决MAC约束问题的不同算法；以及有效的W和Z步骤。PI还将研究这些方法的收敛特性以及它们的并行化能力。机器学习：PI将对几个现有的和新的嵌套模型应用基于MAC的优化：深度神经网络；最佳子集特征选择；决策树的学习特征；字典和分类器学习；参数嵌入；学习哈希函数；远端学习；模型适应；以及其他。评估计划包括标准基准和发音语音建模任务。这项研究是跨学科的，具有潜在的变革性：它为优化和机器学习的研究打开了许多机会，在设计学习系统时促进了MAC公式方面的思维，并可以取代或补充在串行和并行环境下学习嵌套系统的反向传播算法。为发音语音开发的模型也将提高我们对语音产生的理解。

Intelligent processing of complex signals such as images or sound is often performed by a parameterized, nested hierarchy of simple nonlinear processing layers, as in a deep neural net, an object recognition cascade or a speech front-end. Joint estimation of the parameters of all the layers and selection of an optimal architecture is a difficult nonconvex optimization problem, difficult to parallelize, and requiring significant human expert effort, which leads to suboptimal systems in practice. This research will develop a new, general mathematical strategy to learn the parameters and, to some extent, the architecture of nested systems, called the method of auxiliary coordinates (MAC). MAC has provable convergence, is easy to implement reusing existing algorithms for single layers, applies even when parameter derivatives are not available, easily makes use of parallel architectures, and often provides reasonable models within a few iterations. The PI's research and teaching will introduce undergraduate students to the design of nested machine learning systems, and provide computer science graduate students with skills in optimization, in support of specific areas (machine learning, computer vision/speech, etc.). The PI will broadly disseminate the results of the research by publishing papers in optimization, machine learning, computer vision and speech. The PI will make Matlab and C/C++ code available for the algorithms developed and use it as teaching aid in his courses on optimization and machine learning. The PI will strive to involve a diverse population of students in the research.MAC could drastically facilitate, by reducing runtime and human effort, the practical design and estimation of complex models currently being developed in data-rich disciplines such as machine learning, computer vision and speech, but also in other areas of engineering and science. It could also obviate the construction of hand-crafted features in trees and other classifiers. It may bring a wide-ranging and timely benefit to society given that serial computation is reaching a plateau and cloud computing is becoming a commodity, and intelligent data processing is finding its way into mainstream devices (phones, cameras, etc.), thanks to increases in computational power and data availability. The research will develop MAC along two aims. Optimization: The PI will explore different ways to define auxiliary coordinates; different algorithms to solve the MAC-constrained problem; and efficient W and Z steps. The PI will also investigate the convergence properties of these approaches and their ability to be parallelized. Machine learning: The PI will apply MAC-based optimization to several existing and new nested models: deep neural nets; best-subset feature selection; learning features for decision trees; dictionary and classifier learning; parametric embeddings; learning hash functions; distal learning; model adaptation; and others. The evaluation plan includes standard benchmarks and articulatory speech modeling tasks. The research is interdisciplinary and potentially transformative: it opens many opportunities for research in optimization and machine learning, promoting thinking in terms of MAC formulations when designing learning systems, and could replace or complement the backpropagation algorithm in learning nested systems both in the serial and parallel settings. The models developed for articulatory speech will also improve our understanding of speech production.