OAC Core: Small: Higher Order Solvers for Training Machine Learning Models

机器学习（ML）技术已经成为从商业企业到工程设计等广泛应用的关键使能技术。这些技术依赖于复杂的模型，这些模型必须经过大量数据的适当训练。这种训练过程采用数学优化的形式，使训练数据的模型输出与已知输出之间的误差最小化。由于模型中有大量的自由度，目标函数的复杂性被最小化，训练数据量大，有效和高效地训练ML模型的过程是机器学习的关键步骤。该项目的目标是开发新的优化技术，在GPU加速器的大规模并行平台上实现它们，在各种ML应用程序的背景下进行验证，以及开发高度优化，健壮且可用的软件工具和库。这些软件工具将专门用于各种ML模型，并整合到常用的软件框架（如TensorFlow）中，从而使它们能够被非常庞大和多样化的用户社区无缝访问。软件的健壮性、性能和可扩展性提供了独特的功能，有可能重新定义机器学习应用程序的现状，支持更复杂的机器学习模型，增强从训练到测试数据的通用性，并显着减少训练时间。在这些智力和更广泛的影响目标的基础上，该项目整合了一系列旨在扩大参与和创造教育机会和内容的活动。其中包括为本科生提供暑期学校，引导他们从事研究工作，在整个学年中为本科生提供研究机会，开发新的教育材料，将学习与实际使用软件相结合，并激发机器学习中的新公式和方法。该项目的技术目标是通过结合新颖的数值方法、统计抽样技术、高度可扩展的并行实现和高效使用gpu来完成的。该项目有以下具体目标：(i)发展非凸问题的二阶牛顿型方法。具体来说，该项目侧重于基于信任域（TR）和立方正则化（CR）的方法，这些方法依赖于对Hessian和Fisher信息矩阵的近似来提供高效的求解器；（ii）开发完整的高阶优化程序（HOOP）工具包，包括无偏和有偏采样Hessians， Fisher矩阵的块对角近似，共轭梯度（CG）和CG- steihaug求解器的高效预条件，以及特定问题的优化；（iii）基于交替方向乘法器（ADMM）和并行矩阵求解器的组合开发高效并行方法，用于具有GPU加速器的可扩展硬件平台，以及将软件集成到TensorFlow中。该软件还将作为容器化的可执行文件提供，可以在客户端以最小的努力实例化，作为可用于构建新的ML应用程序的库，以及作为教育和培训的web可访问服务；（iv）证明了新方法在重要应用类上的有效性，包括大规模半定规划（SDP）的求解，矩阵分解和距离度量学习问题，以及深度神经网络的训练。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning (ML) techniques have emerged as a key enabling technology for a broad class of applications, from business enterprises to engineering design. These techniques rely on complex models that must be suitably trained on large amounts of data. This training process takes the form of mathematical optimization, which minimizes the error between model output and known output, for training data. Owing to the large number of degrees of freedom in the model, the complexity of the objective function being minimized, and the volume of training data, the process of effectively and efficiently training ML models is a critical step in machine learning. The goal of this project is to develop novel optimization techniques, their implementations on large scale parallel platforms with GPU accelerators, validation in the context of diverse ML applications, and development of highly optimized, robust, and usable software tools and libraries. These software tools will be specialized to various ML models, and incorporated into commonly used software frameworks such as TensorFlow -- thus making them seamlessly accessible to a very large and diverse user community. The robustness, performance, and scalability of the software provide unique capabilities, with the potential to redefine the state of the art in ML applications, in terms of supporting significantly more complex ML models, enhancing generalizability from training to test data, and significantly reducing training time. Building on these intellectual and broader impact goals, the project integrates a number of activities aimed at broadening participation and creating educational opportunities and content. These include summer schools for undergraduate students to channel them into research careers, providing research opportunities for undergraduates through the school year, development of new educational material that integrates learning with hands-on use of software, and motivating novel formulations and methods in machine learning.The technical goals of the project are accomplished through a combination of novel numerical methods, statistical sampling techniques, highly scalable parallel implementations, and efficient use of GPUs. The project has the following specific aims: (i) development of second order Newton-type methods for non-convex problems. Specifically, the project focuses on Trust Region (TR) and Cubic Regularization (CR) based methods that rely on approximations to the Hessian and Fisher information matrices to deliver highly efficient solvers; (ii) development of a complete Higher Order Optimization Procedures (HOOP) toolkit, including unbiased and biased sampled Hessians, block diagonal approximations of the Fisher matrix, efficient and effective preconditioners for the Conjugate Gradient (CG) and CG-Steihaug solvers, and problem-specific optimizations; (iii) development of efficient parallel methods based on a combination of Alternating Direction Method of Multipliers (ADMM) and parallel matrix solvers, for scalable hardware platforms with GPU accelerators, as well as an integration of the software into TensorFlow. The software will also be made available as containerized executables that can be instantiated at clients with minimal effort, as libraries that can be used to build new ML applications, and as web accessible services for education and training; and (iv) demonstration of the effectiveness of the new methods on important application classes, including solution of large-scale semi-definite programs (SDP), problems in matrix factorization and distance metric learning, and training of deep neural networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.