OAC Core: Small: Higher Order Solvers for Training Machine Learning Models

机器学习（ML）技术已经成为从商业企业到工程设计等广泛应用的关键技术。这些技术依赖于复杂的模型，这些模型必须在大量数据上进行适当的训练。该训练过程采用数学优化的形式，其针对训练数据最小化模型输出和已知输出之间的误差。由于模型中的自由度很大，目标函数的复杂性被最小化，以及训练数据的量，有效和高效地训练ML模型的过程是机器学习中的关键步骤。 该项目的目标是开发新的优化技术，在具有GPU加速器的大规模并行平台上实现，在各种ML应用程序的背景下进行验证，以及开发高度优化，强大和可用的软件工具和库。这些软件工具将专门用于各种ML模型，并集成到常用的软件框架中，例如TensorFlow，从而使它们能够无缝地访问非常庞大且多样化的用户社区。 该软件的鲁棒性、性能和可扩展性提供了独特的功能，有可能重新定义ML应用程序的最新技术，支持更复杂的ML模型，增强从训练到测试数据的可推广性，并显着减少训练时间。在这些知识和更广泛影响目标的基础上，该项目整合了一些旨在扩大参与和创造教育机会和内容的活动。 这些措施包括暑期学校为本科生引导他们进入研究生涯，通过学年为本科生提供研究机会，开发新的教育材料，将学习与实际使用软件相结合，并激励机器学习中的新公式和方法。该项目的技术目标通过结合新颖的数值方法，统计抽样技术，高度可扩展的并行实现，以及GPU的高效使用。该项目有以下具体目标：（一）发展非凸问题的二阶牛顿型方法。具体而言，该项目侧重于信赖域（TR）和三次正则化（CR）的方法，依赖于近似的海森和费舍尔信息矩阵，以提供高效的求解器; ㈡开发一套完整的高阶优化程序工具包，包括无偏和有偏抽样海森、费歇尔矩阵的块对角近似法，用于共轭梯度（CG）和CG-Steihaug求解器的高效和有效的预处理器，以及特定问题的优化;（iii）发展以交替方向乘法（ADMM）和并行矩阵求解器相结合为基础的有效并行方法，用于具有GPU加速器的可扩展硬件平台，以及将软件集成到TensorFlow中。该软件还将作为容器化的可执行文件提供，可以在客户端以最小的努力进行实例化，作为可用于构建新ML应用程序的库，以及作为教育和培训的Web访问服务;以及（iv）新方法在重要应用类别上的有效性的示范，包括大规模半定规划（SDP）的解，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning (ML) techniques have emerged as a key enabling technology for a broad class of applications, from business enterprises to engineering design. These techniques rely on complex models that must be suitably trained on large amounts of data. This training process takes the form of mathematical optimization, which minimizes the error between model output and known output, for training data. Owing to the large number of degrees of freedom in the model, the complexity of the objective function being minimized, and the volume of training data, the process of effectively and efficiently training ML models is a critical step in machine learning. The goal of this project is to develop novel optimization techniques, their implementations on large scale parallel platforms with GPU accelerators, validation in the context of diverse ML applications, and development of highly optimized, robust, and usable software tools and libraries. These software tools will be specialized to various ML models, and incorporated into commonly used software frameworks such as TensorFlow -- thus making them seamlessly accessible to a very large and diverse user community. The robustness, performance, and scalability of the software provide unique capabilities, with the potential to redefine the state of the art in ML applications, in terms of supporting significantly more complex ML models, enhancing generalizability from training to test data, and significantly reducing training time. Building on these intellectual and broader impact goals, the project integrates a number of activities aimed at broadening participation and creating educational opportunities and content. These include summer schools for undergraduate students to channel them into research careers, providing research opportunities for undergraduates through the school year, development of new educational material that integrates learning with hands-on use of software, and motivating novel formulations and methods in machine learning.The technical goals of the project are accomplished through a combination of novel numerical methods, statistical sampling techniques, highly scalable parallel implementations, and efficient use of GPUs. The project has the following specific aims: (i) development of second order Newton-type methods for non-convex problems. Specifically, the project focuses on Trust Region (TR) and Cubic Regularization (CR) based methods that rely on approximations to the Hessian and Fisher information matrices to deliver highly efficient solvers; (ii) development of a complete Higher Order Optimization Procedures (HOOP) toolkit, including unbiased and biased sampled Hessians, block diagonal approximations of the Fisher matrix, efficient and effective preconditioners for the Conjugate Gradient (CG) and CG-Steihaug solvers, and problem-specific optimizations; (iii) development of efficient parallel methods based on a combination of Alternating Direction Method of Multipliers (ADMM) and parallel matrix solvers, for scalable hardware platforms with GPU accelerators, as well as an integration of the software into TensorFlow. The software will also be made available as containerized executables that can be instantiated at clients with minimal effort, as libraries that can be used to build new ML applications, and as web accessible services for education and training; and (iv) demonstration of the effectiveness of the new methods on important application classes, including solution of large-scale semi-definite programs (SDP), problems in matrix factorization and distance metric learning, and training of deep neural networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.