Collaborative Research: Algorithms for Large-scale Stochastic and Nonlinear Optimization

几十年来，人工智能的前景一直是公众和私人都感兴趣的话题。从20世纪90年代开始，该领域受益于机器学习领域的快速发展和扩展。从机器学习中产生的智能系统，如搜索引擎、推荐平台、语音和图像识别软件，已经成为现代社会不可或缺的一部分。机器学习技术植根于统计学，严重依赖于数值算法的效率，它利用了日益强大的计算平台和非常大的数据集的可用性。机器学习的支柱之一是数学优化，在这种情况下，数学优化涉及系统参数的计算，该系统旨在根据尚未见过的数据做出决策。该项目的目标是开发新的优化算法，使机器学习领域的持续发展成为可能。该研究包括两个项目，它们在主题上相关，并解决非线性，高维，随机，涉及非常大的数据集，在某些情况下是非凸的优化问题的解决方案。将开发两类算法，以获得随机梯度方法和批处理方法的优点，同时避免它们的缺点。其中一种算法使用梯度聚合方法，该方法重用在以前的迭代中计算的梯度值。我们面临的挑战是设计一种算法，能够有效地减少测试误差，而不仅仅是训练误差。第二种方法采用自适应采样技术，在优化过程中降低随机梯度近似中的噪声。本研究的一个重要方面是设计一种有效的策略，以结合二阶信息，捕获优化损失函数的曲率，即使在Hessian估计基于不准确梯度的情况下。在所有情况下，研究的目标是在软件中设计和实现算法，并在现实的机器学习应用程序中测试它们。

The promise of artificial intelligence has been a topic of both public and private interest for decades. Starting in the 1990s the field has been benefited from the rapidly evolving and expanding field of machine learning. The intelligent systems that have been borne out of machine learning, such as search engines, recommendation platforms, and speech and image recognition software, have become an indispensable part of modern society. Rooted in statistics and relying heavily on the efficiency of numerical algorithms, machine learning techniques capitalize on increasingly powerful computing platforms and the availability of very large datasets. One of the pillars of machine learning is mathematical optimization, which, in this context, involves the computation of parameters for a system designed to make decisions based on yet unseen data. The goal of this project is to develop new optimization algorithms that will enable the continuing rise of the field of machine learning. The research consists of two projects, which are thematically related and address the solution of optimization problems that are nonlinear, high dimensional, stochastic, involve very large data sets and in some cases are non-convex. Two families of algorithms will be developed to garner the benefits of both stochastic gradient methods and batch methods, while avoiding their shortcomings. One of these algorithms uses a gradient aggregation approach that re-uses gradient values computed at previous iterations. The challenge is to design an algorithm that is efficient in minimizing testing error, not just training error. The second approach employs adaptive sampling techniques to reduce the noise in stochastic gradient approximations as the optimization progresses. An important aspect of this research is the design of an efficient strategy for incorporating second-order information that captures curvature of the optimized loss function, even in the case when Hessian estimates are based on inaccurate gradients. In all cases, the goal is research is to design and implement algorithms in software, and test them on realistic machine learning applications.