Parallel Software for Million-scale Exact Kernel Regression
Parallel Software for Million-scale Exact Kernel Regression
复制标题
DOI：
10.1145/3577193.3593737
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Yu Chen;Lucca Skon;James R. McCombs;Zhenming Liu;A. Stathopoulos
中科院分区：
文献类型：
--
作者：
Yu Chen;Lucca Skon;James R. McCombs;Zhenming Liu;A. Stathopoulos
我们提出了一个核主成分回归软件的设计和实现，该软件处理具有一百万或更多观测值的训练数据集。核回归是一种非线性和可解释的模型，具有广泛的下游应用，并且与深度学习有着密切的联系。然而，使用当前可用的软件对大规模内核模型进行精确的回归是出了名的困难，因为它需要大量的计算和内存，并且需要大量的超参数调优。虽然在计算科学中，分布式计算和迭代方法已经成为大规模软件的支柱，但它们尚未被广泛应用于核学习。我们的软件利用现有的高性能计算（HPC）技术，并开发新的解决HPC和学习算法之间的横切约束。它集成了三个主要组件：(a)一个最先进的并行特征值迭代求解器，(b)一个块矩阵向量乘法例程，它采用多线程和分布式内存并行性，可以在有限的内存下实时执行，以及(c)一个由Python前端组成的软件管道，它通过一个提升优化器控制HPC主干和超参数优化。我们通过运行整个ImageNet数据集和一个大型资产定价数据集来执行可行性研究。
We present the design and the implementation of a kernel principal component regression software that handles training datasets with a million or more observations. Kernel regressions are nonlinear and interpretable models that have wide downstream applications, and are shown to have a close connection to deep learning. Nevertheless, the exact regression of large-scale kernel models using currently available software has been notoriously difficult because it is both compute and memory intensive and it requires extensive tuning of hyperparameters. While in computational science distributed computing and iterative methods have been a mainstay of large scale software, they have not been widely adopted in kernel learning. Our software leverages existing high performance computing (HPC) techniques and develops new ones that address cross-cutting constraints between HPC and learning algorithms. It integrates three major components: (a) a state-of-the-art parallel eigenvalue iterative solver, (b) a block matrix-vector multiplication routine that employs both multi-threading and distributed memory parallelism and can be performed on-the-fly under limited memory, and (c) a software pipeline consisting of Python front-ends that control the HPC backbone and the hyperparameter optimization through a boosting optimizer. We perform feasibility studies by running the entire ImageNet dataset and a large asset pricing dataset.