Incremental online learning in high dimensions
Incremental online learning in high dimensions
复制标题
DOI：
10.1162/089976605774320557
复制
发表时间：
2005-12-01
影响因子：
2.9
通讯作者：
Schaal, S
中科院分区：
文献类型：
--
作者：
Vijayakumar, S;D'Souza, A;Schaal, S
局部加权投影回归(LWPR)是一种在具有冗余和无关输入维度的高维空间中增量逼近非线性函数的新算法。它的核心是使用局部线性模型的非参数回归。为了保持计算效率和数值健壮性，每个局部模型本着偏最小二乘回归的精神，在输入空间的选定方向上用少量的单变量回归进行回归分析。我们讨论了局部学习技术何时以及如何在高维空间中成功地工作，并回顾了各种局部降维技术，最后得出了LWPR算法。LWPR的特点是：(1)使用基于增量训练的二阶学习方法快速学习；(2)使用统计上可靠的随机留一交叉验证进行学习，而不需要记忆训练数据；(3)仅基于局部信息调整其加权核，以最小化增量学习的负面干扰的危险；(4)具有输入数量线性的计算复杂性；以及(5)可以处理大量-可能是冗余的-输入，如各种经验评估所示，高达90维的数据集。对于概率解释，推导了预测方差和置信度区间。据我们所知，LWPR是第一种真正的增量式空间局部化学习方法，可以成功和高效地在非常高维的空间中运行。
Locally weighted projection regression (LWPR) is a new algorithm for incremental nonlinear function approximation in high-dimensional spaces with redundant and irrelevant input dimensions. At its core, it employs nonparametric regression with locally linear models. In order to stay computationally efficient and numerically robust, each local model performs the regression analysis with a small number of univariate regressions in selected directions in input space in the spirit of partial least squares regression. We discuss when and how local learning techniques can successfully work in high-dimensional spaces and review the various techniques for local dimensionality reduction before finally deriving the LWPR algorithm. The properties of LWPR are that it (1) learns rapidly with second-order learning methods based on incremental training, (2) uses statistically sound stochastic leave-one-out cross validation for learning without the need to memorize training data, (3) adjusts its weighting kernels based on only local information in order to minimize the danger of negative interference of incremental learning, (4) has a computational complexity that is linear in the number of inputs, and (5) can deal with a large number of-possibly redundant-inputs, as shown in various empirical evaluations with up to 90 dimensional data sets. For a probabilistic interpretation, predictive variance and confidence intervals are derived. To our knowledge, LWPR is the first truly incremental spatially localized learning method that can successfully and efficiently operate in very high-dimensional spaces.