CAREER: Parameter-free Optimization Algorithms for Machine Learning

机器学习（ML）被认为是下一次工业革命的燃料。然而，目前最先进的ML算法仍然严重依赖于人类参与循环才能正常工作。事实上，ML算法的训练过程需要大量的人为干预，通过扭曲和调整算法的许多旋钮。此外，大多数时候，这种调整过程是在没有任何理论指导的情况下进行的。ML从业者最常见的选择是遵循他们的直觉或彻底评估所有可能性，这使得训练这些算法所需的总时间难以预测。在这个项目中，研究人员的目标是为ML设计真正无参数的训练算法，以消除人类参数调整的负担，并使ML算法更具可扩展性。研究人员还提出了一个教育计划，旨在填补大多数学生心目中理论和应用ML之间的差距。它将涉及与青年科学家数学项目（PROMYS）的合作，向高中学生和教师介绍机器学习理论的基本概念，并与女子史密斯学院合作。该项目源于研究人员最近推出的设计最佳无参数优化算法的新方法，例如，无学习率的随机梯度下降，适用于凸函数的特殊情况。在这个项目中，研究人员提出超越凸函数。 特别是，研究者计划追求以下相互关联的目标：1。没有先验知识的学习的计算复杂性。我们的目标是充分表征学习的计算复杂性，而不假设未知量的知识，在最坏的和“简单”的情况下。换句话说，如果一个学习算法需要参数来实现最优性能，那么目标就是充分表征整个计算复杂度，包括搜索最优参数。2.无参数非凸随机优化。该项目的目标是推广最近的无参数优化结果，以处理现代ML算法中的非凸问题。研究人员计划使用一种新的优化框架，该框架混合了在线镜像下降与双重平均的元素，这两个框架用于设计优化算法。减少深度学习优化中的参数。深度学习模型是ML中参数最多的模型。该项目考虑了该项目研究在广泛使用的深度学习算法中的三个应用：自适应缩放算法、鞍点优化和高斯过程贝叶斯优化。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine Learning (ML) has been described as the fuel of the next industrial revolution. Yet, current state-of-the-art ML algorithms still heavily rely on having a human in the loop in order to work properly. Indeed, the training process of a ML algorithm requires significant human intervention through twisting and tuning of the many knobs of the algorithm. Moreover, most of the time this tweaking process is carried out without any theoretical guidelines. The most common options for the ML practitioners are to follow their intuitions or to exhaustively evaluate all the possibilities, making the overall time needed to train these algorithms difficult to predict. In this project, the investigator aims at designing truly parameter-free training algorithms for ML, to remove the burden of human parameter-tuning and make ML algorithms more scalable. The investigator also proposes an Education Plan that is designed to fill the gap between theoretical and applied ML in the minds of most students. It will involve a collaboration with the PROgram in Mathematics for Young Scientists (PROMYS), to introduce the study of very basic concepts of the theory of machine learning to both high school students and teachers, and with the women’s Smith College.The project stems from new methods the investigator has recently introduced to design optimal parameter-free optimization algorithms, e.g., Stochastic Gradient Descent without learning rates, for the particular case of convex functions. In this project, the investigator proposes to go beyond convex functions. In particular, the investigator plans to pursue the following inter-related aims:1. Computational complexity of learning without prior knowledge. The objective is to fully characterize the computational complexity of learning, without assuming knowledge of unknown quantities, in worst and "easy" cases. In other words, if a learning algorithm needs parameters to achieve optimal performance, the goal is to fully characterize the overall computational complexity, including the search for the optimal parameters.2. Non-convex stochastic optimization without parameters. The project's goal is to generalize recent results on parameter-free optimization to deal with the non-convex problems in modern ML algorithms. The investigator plans to use a novel optimization framework that mixes elements of Online Mirror Descent with Dual Averaging, the two frameworks used to design optimization algorithms.3. Reducing parameters in optimization for Deep Learning. Deep Learning models are the most parameter-heavy in ML. The project considers three applications of the project's research to widely used deep-learning algorithms: adaptive scaling heuristics, saddle-point optimization, and Bayesian Optimization with Gaussian Processes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.