Collaborative Research: TRIPODS Institute for Optimization and Learning

这个第一阶段的项目成立了一个NSF TRIPODS研究所，该研究所位于理哈伊大学，与石溪大学和西北大学合作，重点研究机器学习应用工具的新进展。机器学习的一个关键组成部分是数学优化，即使用历史数据来训练工具，以做出未来的预测和决策。传统上，机器学习的优化技术主要集中在简化模型和算法上。然而，最近机器学习工具的革命性飞跃——例如。在许多情况下，由于转向使用更复杂的技术（通常涉及深度神经网络），这些技术得以成为可能。这些技术的持续发展需要统计学家、计算机科学家和应用数学家共同努力，开发更复杂的模型和算法，以及支持它们使用的更全面的理论保证。除了研究目标之外，该研究所还培养统计学、计算机科学和应用数学方面的博士生和博士后，并举办跨学科研讨会和冬/暑期学校。第一阶段的研究工作是分析非凸机器学习模型，设计用于训练它们的优化算法，以及开发非参数模型和相关算法。重点是深度神经网络（dnn），主要是一般的，但也涉及感兴趣的特定架构。该研究所的研究工作强调需要在训练深度神经网络的最先进方法和统计性能保证（例如，泛化误差）之间建立联系，这一点目前还没有得到很好的理解。优化算法的开发以二阶导数型技术为中心，包括（无黑森）牛顿、准牛顿、高斯牛顿及其有限内存变体。最近在设计这些方法方面取得了进展；pi的工作建立在这些努力的基础上，他们在这些方法的设计和实现（包括并行和分布式计算环境）方面拥有广泛的专业知识。非参数模型的发展有望将机器学习方法从大量用户定义参数（例如，定义网络结构或优化算法的学习率）所施加的限制中解放出来。这样的模型可能会导致机器学习的巨大进步，该研究所在这一领域的工作也借鉴了pi在无导数优化方法方面的专业知识，这是非参数设置训练所需要的。在这个TRIPODS研究所，pi用统计学、计算机科学和应用数学这三个学科的统一视角来处理所有这些研究方向。事实上，由于机器学习从这些领域汲取了如此多的经验，未来的进展需要优化专家、学习理论家和统计学家之间的密切合作——到目前为止，这些研究人员群体倾向于使用不同的术语和出版场所单独运作。该研究所以深度学习为重点，旨在促进校际和跨学科合作，克服这些障碍。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This Phase I project forms an NSF TRIPODS Institute, based at Lehigh University and in collaboration with Stony Brook and Northwestern Universities, with a focus on new advances in tools for machine learning applications. A critical component for machine learning is mathematical optimization, where one uses historical data to train tools for making future predictions and decisions. Traditionally, optimization techniques for machine learning have focused on simplified models and algorithms. However, recent revolutionary leaps in the successes of machine learning tools---e.g., for image and speech recognition---have in many cases been made possible by a shift toward using more complicated techniques, often involving deep neural networks. Continued advances in the use of such techniques require combined efforts between statisticians, computer scientists, and applied mathematicians to develop more sophisticated models and algorithms along with more comprehensive theoretical guarantees that support their use. In addition to its research goals, the institute trains Ph.D. students and postdoctoral fellows in statistics, computer science, and applied mathematics, and hosts interdisciplinary workshops and Winter/Summer schools. The research efforts in Phase I are on the analysis of nonconvex machine learning models, the design of optimization algorithms for training them, and on the development of nonparametric models and associated algorithms. The focus is on deep neural networks (DNNs), mostly in general, but also with respect to specific architectures of interest. The institute's research efforts emphasize the need to develop connections between state-of-the-art approaches for training DNNs and statistical performance guarantees (e.g., on generalization errors), which are currently not well understood. Optimization algorithms development centers on second-order-derivative-type techniques, including (Hessian-free) Newton, quasi-Newton, Gauss-Newton, and their limited memory variants. Recent advances have been made in the design of such methods; the PIs' work builds upon these efforts with their broad expertise in the design and implementation (including in parallel and distributed computing environments) of such methods. The development of nonparametric models promises to free machine learning approaches from restrictions imposed by large numbers of user-defined parameters (e.g., defining a network structure or learning rate of an optimization algorithm). Such models could lead to great advances in machine learning, and the institute's work in this area also draws on the PIs expertise in derivative-free optimization methods, which are needed for training in nonparametric settings.In this TRIPODS institute, the PIs approach all of these research directions with a unified perspective in the three disciplines of statistics, computer science, and applied mathematics. Indeed, as machine learning draws so heavily from these areas, future progress requires close collaborations between optimization experts, learning theorists, and statisticians---communities of researchers that, as yet, have tended to operate separately with differing terminology and publication venues. With an emphasis on deep learning, this institute aims to foster intercollegiate and interdisciplinary collaborations that overcome these hindrances.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.