SHF: Small: Algorithms and Software for Scalable Kernel Methods

科学家和工程师对在无法在单个工作站上处理的大型数据集上使用机器学习方法越来越感兴趣。 与此同时，公共和私营机构正在对配备数千个前沿处理器和网络连接的高性能计算（HPC）集群进行大量投资。然而，尽管这种HPC系统的可用性，数据分析任务大多限于单个或几个工作站。原因是，除了少数例外，现有的机器学习软件无法在HPC系统上有效扩展。目前的软件无法满足现场处理大型科学和工程数据集的需要，需要进行大量的下采样，以便使用现有的工具。当前人工智能（AI）工作流程中的一个严重瓶颈是大规模问题的训练成本高昂。现有方法的缓慢收敛和大量的校准超参数（学习率，批量大小和其他控制AI系统性能的旋钮）使得训练非常昂贵。设计和分析用于更快训练的可扩展优化算法，即机器学习（ML）模型参数与数据的拟合，需要用于真实的时间和规模的分析，这是本项目的目标。拟议的研究将介绍新的数值方法和并行算法的二阶/牛顿方法，将量身定制的机器学习（ML）模型，并且将比现有的最先进的方法（一阶方法，如最速下降法）快许多数量级。研究人员计划设计，分析和实现协方差矩阵的鲁棒近似，协方差矩阵是人工智能和计算统计中的一类矩阵，用于统计分析（例如，抽样、风险评估和不确定性量化）。研究人员计划在高性能计算的背景下设计，分析和实现可扩展的快速算法，用于所谓的最近邻问题，这是ML，数据分析和信息检索中的一种特殊方法。由此产生的软件库将为发现和创新提供端到端工具，并为NSF XSEDE基础设施项目提供新的功能。沿着研究活动，一个教育和传播计划被设计用来向学生和研究人员，以及更广泛的计算和应用科学家的观众传达这项工作的结果。这个奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Scientists and engineers are increasingly interested in using machine learning methods on huge datasets that cannot be processed on a single workstation. At the same time public and private institutions are making significant investments on high-performance computing (HPC) clusters equipped with thousands of leading edge processors and network connectivity. However, despite the availability of such HPC systems, data analysis tasks are mostly restricted to a single or a few workstations. The reason is that, with few exceptions, existing machine learning software does not scale efficiently on HPC systems. The need to process in-situ large scientific and engineering datasets is not met with current software and significant downsampling is required in order to use existing tools. A serious bottleneck in current artificial intelligence (AI) workflows is the significant cost of training for large scale problems. The slow convergence of existing methods and the large number of calibration hyper-parameters (learning rate, batch size, and other knobs that control the performance of the AI system) make training extremely expensive. Design and analysis of scalable optimization algorithms for faster training, that is the fitting of the machine learning (ML) model parameters to the data, are needed for analytics in real time and at scale, which is the goal of this project.The proposed research will introduce novel numerical methods and parallel algorithms for second-order/Newton methods that will be tailored to machine learning (ML) models and will be many orders of magnitude faster than the existing state of-the-art (first-order methods like steepest descent). The researchers plan to design, analyze, and implement robust approximations for covariance matrices, a class of matrices in AI and computational statistics, used in statistical analysis (e.g., sampling, risk assessment, and uncertainty quantification). The investigators plan to design, analyze, and implement scalable fast algorithms in the context of high-performance computing for the so called nearest-neighbor problem, a particular method in ML, data analysis, and information retrieval. The resulting software library will provide a means for end-to-end tools for discovery and innovation and provide new capabilities in the NSF XSEDE infrastructure project. Along with research activities, an educational and dissemination program is designed to communicate the results of this work to both students and researchers, as well as a more general audience of computational and application scientists.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.