Collaborative Research: AF: Small: A Unified Framework for Analyzing Adaptive Stochastic Optimization Methods Based on Probabilistic Oracles

数据科学和机器学习已经改变了现代科学、工程和商业。 现代机器学习技术的支柱之一是数学优化，这是一种驱动从可用和/或实时生成的数据中学习的方法。 然而，不幸的是，尽管某些优化技术取得了成功，但大规模学习在时间和精力方面仍然非常昂贵，这使得训练机器执行某些基本任务的能力完全掌握在那些能够访问极端规模超级计算设施的人手中。 许多当代技术的一个显著缺陷是，它们以规定的“轨迹“”启动”算法，尽管算法将遵循的实际轨迹取决于未知因素。 当代机器学习的优化技术基本上是通过“调整”算法参数来解决这一问题的，这意味着目标通常只有在多次昂贵的失误之后才能被击中。 当代技术的另一个显著缺陷是经常对正在执行的优化进行限制性假设，这通常包括机器学习模型正在使用未损坏的数据进行训练的假设。 现代现实世界的应用程序要复杂得多。这个项目将探索机器学习和相关主题的自适应（“自调整”）优化技术的设计和分析。 一个目标是产生具有严格保证的自适应算法，该算法可以避免当代算法用于参数调整所需的浪费计算的极端量。 另一个目标是将这些算法的使用扩展到具有不完美的数据/信息的设置，这可能是由于有偏的函数信息、损坏的数据或用于近似目标的新技术。 最后，许多应用最终要求学习过程或模型满足某些显式或隐式约束。 这种机器学习应用的优化方法仍处于起步阶段，主要是由于其更复杂的性质和对算法参数的进一步依赖。 该项目旨在设计一个分析自适应随机优化方法的统一框架，为研究人员和从业者提供一套易于使用的工具，用于设计用于尖端应用的下一代算法。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data science and machine learning have transformed modern science, engineering, and business. One of the pillars of modern-day machine-learning technology is mathematical optimization, which is the methodology that drives the process of learning from available and/or real-time generated data. Unfortunately, however, despite the successes of certain optimization techniques, large-scale learning remains extremely expensive in terms of time and energy, which puts the ability to train machines to perform certain fundamental tasks exclusively in the hands of those with access to extreme-scale supercomputing facilities. A significant deficiency of many contemporary techniques is that they "launch" an algorithm with a prescribed "trajectory," despite the fact that the actual trajectory that the algorithm will follow depends on unknown factors. Contemporary optimization techniques for machine learning essentially account for this by "tuning" algorithmic parameters, which means that the target is typically only hit after numerous expensive misses. Another significant deficiency of contemporary techniques is the restrictive set of assumptions often made about the optimization being performed, which typically includes the assumption that the machine-learning model is being trained with uncorrupted data. Modern real-world applications are far more complex.This project will explore the design and analysis of adaptive ("self-tuning") optimization techniques for machine learning and related topics. One goal is to produce adaptive algorithms with rigorous guarantees that can avoid the extreme amounts of wasteful computation that are required by contemporary algorithms for parameter tuning. Another goal is to extend the use of these algorithms to settings with imperfect data/information, which may be due to biased function information, corrupted data, or novel techniques for approximating the objective. Finally, many applications ultimately require the learning process or model to satisfy some explicit or implicit constraints. Optimization methods for such machine-learning applications are still in their infancy, largely due to their more complicated nature and further dependence on algorithmic parameters. This project aims to design a unified framework for analyzing adaptive stochastic optimization methods that will offer researchers and practitioners a set of easy-to-use tools for designing next-generation algorithms for cutting-edge applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.