Collaborative Research: Algorithms for Large-scale Stochastic and Nonlinear Optimization

合作研究：大规模随机和非线性优化算法

• 批准号: 1620070
• 负责人: Richard Byrd
• 金额: $ 13.64万
• 依托单位: University of Colorado at Boulder
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-08-01 至 2020-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1620070&HistoricalAwards=false
• 关键词: Collaborative; Research; Algorithms; Large; scale

The promise of artificial intelligence has been a topic of both public and private interest for decades. Starting in the 1990s the field has been benefited from the rapidly evolving and expanding field of machine learning. The intelligent systems that have been borne out of machine learning, such as search engines, recommendation platforms, and speech and image recognition software, have become an indispensable part of modern society. Rooted in statistics and relying heavily on the efficiency of numerical algorithms, machine learning techniques capitalize on increasingly powerful computing platforms and the availability of very large datasets. One of the pillars of machine learning is mathematical optimization, which, in this context, involves the computation of parameters for a system designed to make decisions based on yet unseen data. The goal of this project is to develop new optimization algorithms that will enable the continuing rise of the field of machine learning. The research consists of two projects, which are thematically related and address the solution of optimization problems that are nonlinear, high dimensional, stochastic, involve very large data sets and in some cases are non-convex. Two families of algorithms will be developed to garner the benefits of both stochastic gradient methods and batch methods, while avoiding their shortcomings. One of these algorithms uses a gradient aggregation approach that re-uses gradient values computed at previous iterations. The challenge is to design an algorithm that is efficient in minimizing testing error, not just training error. The second approach employs adaptive sampling techniques to reduce the noise in stochastic gradient approximations as the optimization progresses. An important aspect of this research is the design of an efficient strategy for incorporating second-order information that captures curvature of the optimized loss function, even in the case when Hessian estimates are based on inaccurate gradients. In all cases, the goal is research is to design and implement algorithms in software, and test them on realistic machine learning applications.

几十年来，人工智能的前景一直是公共和私人感兴趣的话题。从20世纪90年代开始，该领域受益于快速发展和扩大的机器学习领域。从机器学习中诞生的智能系统，如搜索引擎、推荐平台、语音和图像识别软件，已经成为现代社会不可或缺的一部分。机器学习技术植根于统计学，在很大程度上依赖于数值算法的效率，它利用了日益强大的计算平台和非常大的数据集的可用性。机器学习的支柱之一是数学优化，在这种情况下，它涉及到计算系统的参数，该系统旨在根据尚未见过的数据做出决策。这个项目的目标是开发新的优化算法，使机器学习领域继续崛起。这项研究由两个主题相关的项目组成，它们解决的是非线性、高维、随机的优化问题，涉及非常大的数据集，在某些情况下是非凸的。将开发两类算法，以获得随机梯度方法和批处理方法的优点，同时避免它们的缺点。其中一个算法使用梯度聚合方法，该方法重复使用在先前迭代中计算的梯度值。挑战在于设计一种有效地最小化测试误差的算法，而不仅仅是训练误差。第二种方法采用自适应采样技术，在优化过程中减少随机梯度近似中的噪声。这项研究的一个重要方面是设计一种有效的策略来结合二阶信息，该策略捕捉优化损失函数的曲率，即使在Hessian估计基于不准确的梯度的情况下也是如此。在所有情况下，研究的目标都是在软件中设计和实现算法，并在现实的机器学习应用程序中测试它们。