Collaborative Research: AF: Small: A Unified Framework for Analyzing Adaptive Stochastic Optimization Methods Based on Probabilistic Oracles

数据科学和机器学习已经改变了现代科学、工程和商业。现代机器学习技术的支柱之一是数学优化，这是一种从可用和/或实时生成的数据中驱动学习过程的方法。然而，不幸的是，尽管某些优化技术取得了成功，大规模学习在时间和精力方面仍然非常昂贵，这使得训练机器执行某些基本任务的能力只掌握在那些拥有超大规模超级计算设施的人手中。许多当代技术的一个显著缺陷是，尽管算法将遵循的实际轨迹取决于未知因素，但它们会按照规定的“轨迹”“启动”算法。机器学习的当代优化技术基本上是通过“调整”算法参数来解决这个问题的，这意味着目标通常只有在多次昂贵的失误之后才会被击中。当代技术的另一个重大缺陷是，通常对正在执行的优化进行限制性假设，其中通常包括假设机器学习模型正在使用未损坏的数据进行训练。现代现实世界的应用程序要复杂得多。该项目将探索机器学习和相关主题的自适应（“自调优”）优化技术的设计和分析。一个目标是产生具有严格保证的自适应算法，以避免当代算法在参数调优时所需要的极端数量的浪费计算。另一个目标是将这些算法的使用扩展到具有不完美数据/信息的设置，这可能是由于有偏差的函数信息，损坏的数据或近似目标的新技术。最后，许多应用程序最终要求学习过程或模型满足一些显式或隐式约束。这类机器学习应用的优化方法仍处于起步阶段，主要是由于它们更复杂的性质和对算法参数的进一步依赖。本项目旨在设计一个统一的框架来分析自适应随机优化方法，为研究人员和实践者提供一套易于使用的工具，为前沿应用设计下一代算法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data science and machine learning have transformed modern science, engineering, and business. One of the pillars of modern-day machine-learning technology is mathematical optimization, which is the methodology that drives the process of learning from available and/or real-time generated data. Unfortunately, however, despite the successes of certain optimization techniques, large-scale learning remains extremely expensive in terms of time and energy, which puts the ability to train machines to perform certain fundamental tasks exclusively in the hands of those with access to extreme-scale supercomputing facilities. A significant deficiency of many contemporary techniques is that they "launch" an algorithm with a prescribed "trajectory," despite the fact that the actual trajectory that the algorithm will follow depends on unknown factors. Contemporary optimization techniques for machine learning essentially account for this by "tuning" algorithmic parameters, which means that the target is typically only hit after numerous expensive misses. Another significant deficiency of contemporary techniques is the restrictive set of assumptions often made about the optimization being performed, which typically includes the assumption that the machine-learning model is being trained with uncorrupted data. Modern real-world applications are far more complex.This project will explore the design and analysis of adaptive ("self-tuning") optimization techniques for machine learning and related topics. One goal is to produce adaptive algorithms with rigorous guarantees that can avoid the extreme amounts of wasteful computation that are required by contemporary algorithms for parameter tuning. Another goal is to extend the use of these algorithms to settings with imperfect data/information, which may be due to biased function information, corrupted data, or novel techniques for approximating the objective. Finally, many applications ultimately require the learning process or model to satisfy some explicit or implicit constraints. Optimization methods for such machine-learning applications are still in their infancy, largely due to their more complicated nature and further dependence on algorithmic parameters. This project aims to design a unified framework for analyzing adaptive stochastic optimization methods that will offer researchers and practitioners a set of easy-to-use tools for designing next-generation algorithms for cutting-edge applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.