Collaborative Research: TRIPODS Institute for Optimization and Learning

这个第一阶段的项目成立了一个NSF三脚架研究所，总部设在利哈伊大学，并与石溪大学和西北大学合作，专注于机器学习应用工具的新进展。机器学习的一个关键组成部分是数学优化，人们使用历史数据来训练工具，以便做出未来的预测和决策。传统上，机器学习的优化技术侧重于简化的模型和算法。然而，最近机器学习工具的成功--例如，用于图像和语音识别--的革命性飞跃，在许多情况下是由于转向使用更复杂的技术，通常涉及深度神经网络。在使用这类技术方面的持续进步需要统计学家、计算机科学家和应用数学家共同努力，以开发更复杂的模型和算法，以及支持它们使用的更全面的理论保证。除了其研究目标外，该研究所还培训统计学、计算机科学和应用数学方面的博士生和博士后研究员，并举办跨学科研讨会和冬季/夏季学校。第一阶段的研究工作集中在非凸机器学习模型的分析、训练它们的优化算法的设计以及非参数模型及其相关算法的开发上。重点放在深度神经网络(DNN)上，主要是一般性的，但也涉及到感兴趣的特定体系结构。该研究所的研究工作强调，需要在目前还不能很好理解的最先进的DNN培训方法和统计业绩保证(例如，关于推广误差)之间建立联系。优化算法的开发集中在二阶导数类型的技术上，包括(黑森自由)牛顿、拟牛顿、高斯-牛顿及其有限记忆变体。最近在这种方法的设计方面取得了进展；私人投资机构的工作建立在这些努力的基础上，他们在设计和实施这种方法(包括在并行和分布式计算环境中)方面拥有广泛的专业知识。非参数模型的发展承诺将机器学习方法从大量用户定义的参数(例如，定义网络结构或优化算法的学习速率)施加的限制中解放出来。这样的模型可能会导致机器学习的巨大进步，该研究所在这一领域的工作也借鉴了PI在非参数设置培训中所需的无导数优化方法方面的专业知识。在这个三脚架研究所，PI以统计、计算机科学和应用数学三个学科的统一视角处理所有这些研究方向。事实上，由于机器学习在这些领域中发挥了如此重要的作用，未来的进步需要优化专家、学习理论家和统计学家之间的密切合作-研究人员社区，到目前为止，他们倾向于以不同的术语和出版场所单独操作。以深度学习为重点，该研究所旨在促进校际和学科间的合作，以克服这些障碍。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This Phase I project forms an NSF TRIPODS Institute, based at Lehigh University and in collaboration with Stony Brook and Northwestern Universities, with a focus on new advances in tools for machine learning applications. A critical component for machine learning is mathematical optimization, where one uses historical data to train tools for making future predictions and decisions. Traditionally, optimization techniques for machine learning have focused on simplified models and algorithms. However, recent revolutionary leaps in the successes of machine learning tools---e.g., for image and speech recognition---have in many cases been made possible by a shift toward using more complicated techniques, often involving deep neural networks. Continued advances in the use of such techniques require combined efforts between statisticians, computer scientists, and applied mathematicians to develop more sophisticated models and algorithms along with more comprehensive theoretical guarantees that support their use. In addition to its research goals, the institute trains Ph.D. students and postdoctoral fellows in statistics, computer science, and applied mathematics, and hosts interdisciplinary workshops and Winter/Summer schools. The research efforts in Phase I are on the analysis of nonconvex machine learning models, the design of optimization algorithms for training them, and on the development of nonparametric models and associated algorithms. The focus is on deep neural networks (DNNs), mostly in general, but also with respect to specific architectures of interest. The institute's research efforts emphasize the need to develop connections between state-of-the-art approaches for training DNNs and statistical performance guarantees (e.g., on generalization errors), which are currently not well understood. Optimization algorithms development centers on second-order-derivative-type techniques, including (Hessian-free) Newton, quasi-Newton, Gauss-Newton, and their limited memory variants. Recent advances have been made in the design of such methods; the PIs' work builds upon these efforts with their broad expertise in the design and implementation (including in parallel and distributed computing environments) of such methods. The development of nonparametric models promises to free machine learning approaches from restrictions imposed by large numbers of user-defined parameters (e.g., defining a network structure or learning rate of an optimization algorithm). Such models could lead to great advances in machine learning, and the institute's work in this area also draws on the PIs expertise in derivative-free optimization methods, which are needed for training in nonparametric settings.In this TRIPODS institute, the PIs approach all of these research directions with a unified perspective in the three disciplines of statistics, computer science, and applied mathematics. Indeed, as machine learning draws so heavily from these areas, future progress requires close collaborations between optimization experts, learning theorists, and statisticians---communities of researchers that, as yet, have tended to operate separately with differing terminology and publication venues. With an emphasis on deep learning, this institute aims to foster intercollegiate and interdisciplinary collaborations that overcome these hindrances.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.