Collaborative Research: Algorithms for Large-Scale Stochastic and Nonlinear Optimization

几十年来，人工智能的前景一直是公共和私人利益的话题。从20世纪90年代开始，该领域受益于机器学习领域的快速发展和扩展。由机器学习衍生出来的智能系统，如搜索引擎、推荐平台、语音和图像识别软件，已经成为现代社会不可或缺的一部分。 机器学习技术植根于统计学，严重依赖数值算法的效率，利用日益强大的计算平台和非常大的数据集的可用性。 机器学习的支柱之一是数学优化，在这种情况下，它涉及到为系统计算参数，该系统旨在根据尚未看到的数据做出决策。该项目的目标是开发新的优化算法，使机器学习领域的持续增长成为可能。 该研究由两个项目组成，这两个项目在主题上是相关的，并解决非线性，高维，随机，涉及非常大的数据集，在某些情况下是非凸的优化问题的解决方案。将开发两个系列的算法，以获得随机梯度方法和批处理方法的好处，同时避免它们的缺点。这些算法之一使用梯度聚合方法，该方法重新使用在先前迭代中计算的梯度值。挑战在于设计一种有效的算法，使测试误差最小化，而不仅仅是训练误差。第二种方法采用自适应采样技术，以减少随机梯度近似的优化过程中的噪声。这项研究的一个重要方面是设计一个有效的策略，将二阶信息，捕捉曲率的优化损失函数，即使在海森估计是基于不准确的梯度的情况下。在所有情况下，研究的目标都是在软件中设计和实现算法，并在现实的机器学习应用程序中测试它们。

The promise of artificial intelligence has been a topic of both public and private interest for decades. Starting in the 1990s the field has been benefited from the rapidly evolving and expanding field of machine learning. The intelligent systems that have been borne out of machine learning, such as search engines, recommendation platforms, and speech and image recognition software, have become an indispensable part of modern society. Rooted in statistics and relying heavily on the efficiency of numerical algorithms, machine learning techniques capitalize on increasingly powerful computing platforms and the availability of very large datasets. One of the pillars of machine learning is mathematical optimization, which, in this context, involves the computation of parameters for a system designed to make decisions based on yet unseen data. The goal of this project is to develop new optimization algorithms that will enable the continuing rise of the field of machine learning. The research consists of two projects, which are thematically related and address the solution of optimization problems that are nonlinear, high dimensional, stochastic, involve very large data sets and in some cases are non-convex. Two families of algorithms will be developed to garner the benefits of both stochastic gradient methods and batch methods, while avoiding their shortcomings. One of these algorithms uses a gradient aggregation approach that re-uses gradient values computed at previous iterations. The challenge is to design an algorithm that is efficient in minimizing testing error, not just training error. The second approach employs adaptive sampling techniques to reduce the noise in stochastic gradient approximations as the optimization progresses. An important aspect of this research is the design of an efficient strategy for incorporating second-order information that captures curvature of the optimized loss function, even in the case when Hessian estimates are based on inaccurate gradients. In all cases, the goal is research is to design and implement algorithms in software, and test them on realistic machine learning applications.