NSF-BSF: AF: Small: Parameter-Free Stochastic Optimization via Trajectory Cues

机器学习在使计算机能够从数据中学习以执行图像注释、语音识别和语言翻译方面非常成功。尽管取得了这一成功，但许多潜在的应用仍然没有得到开发。更广泛地采用机器学习的一个主要障碍是机器学习方法必须手动调整到每个新的学习任务。这种调整需要高水平的计算资源和用户技能，这排除了许多潜在的机器学习用户。此外，即使是领先的机器学习从业者也会犯代价高昂的调优错误。本项目旨在使调优过程更简单，计算效率更高。本项目的重点是从为机器学习提供动力的随机优化算法中消除参数调优。为了实现这一目标，将采用两种新的策略。首先，该项目将开发用于无噪声优化的算法，而不是使用在线到批量转换（随机优化中的标准技术），然后通过强大的时间均匀测量结果集中将其提升到随机设置。其次，该项目将开发基于轨迹线索的算法：从优化轨迹直接观察到的量（例如迭代与初始点之间的距离），这些量作为未知问题参数的替代。这两个新的策略将使该项目能够解决无参数优化中的突出理论问题，并设计实用的算法，减少参数调整的负担。该项目还将研究针对特定问题类（如深度学习）的无参数方法。这项研究产生的所有软件都将是开源的。该项目的算法将被整合到本科和研究生阶段的优化和机器学习课程中。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning has been extremely successful at enabling computers to learn from data to perform image annotation, speech recognition, and language translation. Despite this success, many potential applications remain untapped. One major barrier to more widespread adoption of machine learning is that machine learning methods must be manually tuned to each new learning task. This tuning requires high levels of both computational resources and user skill, which excludes many potential machine learning users. Furthermore, even leading machine learning practitioners make costly tuning mistakes. This project aims to make the tuning process simpler and more computationally efficient.This project focuses on eliminating parameter tuning from the stochastic optimization algorithms that power machine learning. Toward this goal, two novel strategies will be employed. First, instead of using online-to-batch conversion, a standard technique in stochastic optimization, this project will develop algorithms for noiseless optimization and then lift them to the stochastic setting via powerful time-uniform concentration of measure results. Second, the project will develop algorithms based on trajectory cues: quantities directly observable from the optimization trajectory (such as the distance between iterates and the initial point) that serve as stand-ins for unknown problem parameters. These two new strategies will enable this project to tackle outstanding theoretical problems in parameter-free optimization and design practical algorithms that reduce the burden of parameter tuning. The project will also examine parameter-free methods for particular problem classes such as deep learning. All software resulting from this research will be open source. Algorithms from this project will be integrated into optimization and machine learning curricula at both the undergraduate and graduate level.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.