RI: Medium: Scalable Second-order Methods for Training, Designing, and Deploying Machine Learning Models

能够处理大规模现代数据集的可伸缩优化算法是许多机器学习应用程序的组成部分。仅使用一阶导数信息的优化方法，即所谓的一阶方法，是最常见的。然而，这些一阶方法中的许多都有固有的缺点，例如，收敛速度慢，沟通差，需要费力地手动调整。另一方面，所谓的二阶方法，即使用二阶导数信息的方法，配备了缓解许多这些缺点的能力，但它们在机器学习中的使用要少得多。这个项目将开发、实施和应用新的方法，通过创新地应用二阶信息来增强机器学习模型的设计、诊断和培训。技术工作将侧重于理论发展、有效实施和在多种环境下的应用。理论发展将从四个一般角度解决训练大规模非凸机器学习模型所涉及的挑战：高质量的局部极小值；分布式计算环境；泛化性能；以及加速。这项工作还将开发高效的黑森语诊断工具，用于分析培训过程以及已培训的模型。最后，提出的方法的改进和在各种环境中的应用将被开发：改善通信特性；利用对抗性数据；以及探索如何将这些想法用于更具挑战性的问题，例如如何改进神经结构设计和搜索。在所有情况下，都将为共享内存和分布式计算环境提供高质量的用户友好型实施。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Scalable optimization algorithms that can handle large-scale modern datasets are an integral part of many applications of machine learning. Optimization methods that use only first derivative information, i.e., so-called first-order methods, are most common. However, many of these first-order methods come with inherent disadvantages, e.g., slow convergence, poor communication, and the need for laborious manual tuning. On the other hand, so-called second-order methods, i.e., methods that use second derivative information, come equipped with the ability to mitigate many of these disadvantages, but they are far less used within machine learning. This project will develop, implement, and apply novel methods that by innovative application of second-order information allow for enhanced design, diagnostics, and training of machine learning models. Technical work will focus on theoretical developments, efficient implementations, and applications in multiple settings. Theoretical developments will tackle challenges involved in training large-scale non-convex machine learning models from four general angles: high-quality local minima; distributed computing environments; generalization performance; and acceleration. Work will also develop efficient Hessian-based diagnostics tools for the analysis of the training process as well as of already-trained models. Finally, improvements and applications of the proposed methods in a variety settings will be developed: improved communication properties; exploiting adversarial data; and exploring how these ideas can be used for more challenging problems such as how to improve neural architecture design and search. In all cases, high-quality user-friendly implementations for both shared-memory and distributed computing environments will be made available.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.