EAGER: New Optimization Methods for Machine Learning

该提案通过将机器学习问题的基础简化为更简单的易于解决的更新规则，探索了复杂非线性方程的优化。学习问题包括分类、回归、无监督学习等。通过一种称为优化的方法，复杂的优化问题可以通过迭代求解最小二乘和传统线性代数运算等简单问题来处理。该提案的重点是如何并行化这种方法，以便它可以有效地同时利用许多CPU/GPU，并以分布式方式。此外，通过使该方法随机化，在大型或流式数据集上的更快收敛成为可能。探索其他变化，例如稀疏学习，其中恢复的解决方案被迫紧凑，这也导致进一步的效率。 越来越多地，文献中的绝大多数机器学习问题都是通过使用通用的一阶和二阶方法来优化的。 该提案中的方法是专门为机器学习优化问题设计的，并使用优化和定界来保证单调收敛。 在前期工作中，优化在实践中产生了更快的收敛，以及新的理论保证。为了使该方法在实践中真正可行，该建议提出了分布式，并行，随机和稀疏扩展。由于这种扩展可能会违反单调收敛保证，该提案探讨了额外的算法和理论努力，以保持保证，同时也获得快速算法。特别是，并行化和分布式计算进行包装当前国家的最先进的最小二乘求解器与约束优化步骤。随机计算探索使用单件，小批量和可变大小的批量方法。稀疏性是通过迭代当前的大规模稀疏求解器，如FISTA和QUIC在边界优化技术。 在更广泛的影响方面，一名研究生将得到支持，并将帮助为机器学习专家和从业者制作可下载的工具。 将开发模块，以增加PI现有的机器学习课程。PI将组织为期一天的优化方法研讨会。 该提案还提供了一个公共项目网站，供查阅研究出版物、软件/数据下载和即将举行的活动时间表。

This proposal explores the optimization of complicated nonlinear equations that underlie machine learning problems by reducing them to simpler easy-to-solve update rules. The learning problems include classification, regression, unsupervised learning and more. Through a method known as majorization, complicated optimization problems are handled by iteratively solving simpler problems like least-squares and traditional linear algebra operations. The proposal focuses on how to parallelize this method so that it can efficiently leverage many CPUs/GPUs simultaneously and in a distributed manner. Furthermore, by making the method stochastic, faster convergence on large or streaming data-sets becomes possible. Other variations are explored such as sparse learning where the recovered solution is forced to be compact which also leads to further efficiency. Increasingly, the vast majority of machine learning problems in the literature are optimized by using generic first- and second-order methods. The approach in this proposal is designed specifically for machine learning optimization problems and uses majorization and bounding to guarantee monotonic convergence. In preliminary work, majorization has produced faster convergence in practice as well as novel theoretical guarantees. To make the method truly viable in practice, this proposal puts forward distributed, parallel, stochastic and sparse extensions. Since such extensions may violate monotonic convergence guarantees, the proposal explores additional algorithmic and theoretical efforts to preserve guarantees while also obtaining fast algorithms. In particular, parallelization and distributed computation is performed by wrapping current state-of-the-art least squares solvers with bound majorization steps. Stochastic computation is explored using singleton, small-batch and variable-sized batch methods. Sparsity is achieved by iterating current large-scale sparse solvers like FISTA and QUIC within the bound majorization technique. In terms of broader impact, one graduate student will be supported and will help produce downloadable tools for machine learning experts as well as practitioners. Modules will be developed to add to the PI's existing courses in machine learning. The PI will organize a one-day workshop on majorization methods. The proposal also provides a public project website with access to research publications, software/data downloads and schedules of upcoming events.