Collaborative Research: Robust Acceleration and Preconditioning Methods for Data-Related Applications: Theory and Practice

协作研究：数据相关应用的鲁棒加速和预处理方法：理论与实践

• 批准号: 2208412
• 负责人: Yuanzhe Xi
• 金额: $ 20万
• 依托单位: Emory University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2208412&HistoricalAwards=false
• 关键词: Collaborative; Research; Robust; Acceleration; Preconditioning

In many disciplines of science and engineering one often encounters sequences of numbers or vectors or other mathematical objects. A common goal in these situations is to obtain the limit of the sequence inexpensively. As a simple example, there are several ways to generate a sequence of numbers that converge to the number pi and some sequences will reach the limit pi rather quickly. In some cases, it may be possible to modify the original method that produced the sequence to obtain a faster converging one. However, this is not always possible or cost-effective because the process by which the sequence is produced is not explicit or it may be too cumbersome for this approach to be practical. Another common solution is to transform the sequence, by 'accelerating it'. This usually entails combining the terms of the sequence to produce new entries that reach the limit faster. So far, acceleration methods were developed by mathematicians and (quantum) physicists to deal with a wide range of problems from the physical sciences. The primary goal of this project is to study such acceleration methods and to adapt them to the modern era by focusing on topics related to machine learning and, more generally, data sciences. This collaborative research project between researchers at Emory University and the University of Minnesota, will develop and study theoretically a number of robust acceleration algorithms with an emphasis on problems that stem from data-related applications, as well as train graduate students in this field of study. The need to accelerate numerical sequences of various types has frequently been felt across a wide range of disciplines and it has been addressed by many researchers for quite some time. In the past, such acceleration or extrapolation schemes were targeted mainly toward sequences of vectors that arise from physical simulations, e.g., the sequence of potentials generated by the Self Consistent Field (SCF) iterations in quantum physics. In recent years, the rapid expansion of machine learning methodologies across a great variety of disciplines has generated new demand for algorithms to accelerate sequences of various types. However, the new type of sequences encountered in these applications differ in fundamental ways from their analogues in physical simulations. In contrast with the common setting required in, e.g., quantum physics, calculations in machine learning are often performed in single or half precision instead of double precision. Furthermore, in neural networks these sequences tend to be very irregular because they originate from stochastic gradient approaches. The collaborative team from Emory University and the University of Minnesota, will develop and study theoretically a number of robust acceleration algorithms with an emphasis on their application to irregular sequences such as those encountered in data-related applications. The investigating team will study a number of strategies for improving the robustness of standard acceleration schemes, such as Anderson mixing, or the epsilon algorithm. A number of recently advocated second-order methods, based on (so-called) momentum ideas, have been shown to be of great help in accelerating standard stochastic gradient descent methods in Deep Learning. The investigators will add to these schemes another method of the same class that is grounded in Chebyshev acceleration. One advantage of the Chebyshev-based scheme is that it is fairly easy to study theoretically in part because it is well understood for linear problems. In a second research direction, the investigating team will study acceleration methods in the specific context of machine learning tasks. For example, inspired by recent advances in parameter averaging and variance reduction schemes they will explore the application of classical extrapolation methods to stochastic gradient sequences as an adaptive extrapolation procedure. Experiments show that parameter averaging is key to the success of acceleration procedures. Just as important is the selection of the vectors to accelerate. Along the same lines, robust acceleration schemes will be designed and studied for sequences hampered by low precision arithmetic.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在科学和工程的许多学科中，人们经常会遇到数字序列或向量或其他数学对象。在这些情况下的一个共同目标是廉价地获得序列的极限。举一个简单的例子，有几种方法可以生成一个收敛到pi的数列，有些数列会很快达到pi的极限。在某些情况下，可以修改产生序列的原始方法，以获得更快的收敛。然而，这并不总是可能的或具有成本效益的，因为产生序列的过程并不明确，或者对于这种方法来说可能太麻烦而不实用。另一种常见的解决方案是通过“加速”来转换序列。这通常需要组合序列的项以产生更快达到极限的新条目。到目前为止，加速方法是由数学家和（量子）物理学家开发的，用于处理物理科学中的各种问题。该项目的主要目标是研究这种加速方法，并通过关注与机器学习以及更广泛的数据科学相关的主题来使其适应现代社会。埃默里大学和明尼苏达大学的研究人员之间的这一合作研究项目将在理论上开发和研究一些强大的加速算法，重点是源于数据相关应用的问题，并培养这一研究领域的研究生。加速各种类型的数字序列的需要在广泛的学科中经常被感受到，并且许多研究人员已经解决了相当长的一段时间。在过去，这种加速或外推方案主要针对由物理模拟产生的矢量序列，例如，量子物理学中自洽场（SCF）迭代产生的电势序列。近年来，机器学习方法在各种学科中的快速扩展对加速各种类型序列的算法产生了新的需求。然而，在这些应用中遇到的新类型的序列在根本上不同于它们在物理模拟中的类似物。与例如，在量子物理学中，机器学习中的计算通常以单精度或半精度而不是双精度执行。此外，在神经网络中，这些序列往往是非常不规则的，因为它们源于随机梯度方法。来自埃默里大学和明尼苏达大学的合作团队将在理论上开发和研究一些强大的加速算法，重点是它们在不规则序列中的应用，例如在数据相关应用中遇到的那些。调查小组将研究一些策略，以提高标准的加速计划，如安德森混合，或鲁棒性的算法。最近提出的一些基于（所谓的）动量思想的二阶方法已被证明对加速深度学习中的标准随机梯度下降方法有很大帮助。研究人员将在这些方案中添加另一种基于切比雪夫加速度的同类方法。切比雪夫格式的一个优点是理论上很容易研究，部分原因是它对于线性问题很容易理解。在第二个研究方向中，研究团队将研究机器学习任务特定背景下的加速方法。例如，受参数平均和方差减少方案的最新进展的启发，他们将探索经典外推方法作为自适应外推程序应用于随机梯度序列。实验表明，参数平均是加速过程成功的关键。同样重要的是选择要加速的向量。沿着同样的路线，将设计和研究稳健的加速方案，用于受低精度算法阻碍的序列。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。