Direct Estimation of the Derivative of Quadratic Mutual Information with Application in Supervised Dimension Reduction
Direct Estimation of the Derivative of Quadratic Mutual Information with Application in Supervised Dimension Reduction
复制标题
DOI：
10.1162/neco_a_00986
复制
发表时间：
2015-08
影响因子：
2.9
通讯作者：
Voot Tangkaratt;Hiroaki Sasaki;Masashi Sugiyama
中科院分区：
文献类型：
--
作者：
Voot Tangkaratt;Hiroaki Sasaki;Masashi Sugiyama
摘要线性监督降维的一个典型目标是找到输入空间的一个低维子空间，使得投影输入变量保留关于输出变量的最大信息。依赖最大化方法通过最大化投影输入变量和输出变量之间的统计依赖来解决监督降维问题。一个著名的统计相关性度量是互信息（MI），它基于Kullback-Leibler（KL）散度。然而，已知KL发散对异常值敏感。二次MI（QMI）是基于距离的MI的一种变体，它比KL散度更能抵抗离群值，最近提出了一种从数据中估计QMI的计算效率高的方法--最小二乘QMI（LSQMI）。基于这些原因，开发一种基于LSQMI的监督降维方法似乎很有希望。然而，线性监督降维中的子空间搜索需要的不是QMI本身，而是QMI的导数，并且准确的QMI估计量的导数不一定是QMI导数的良好估计量。在这封信中，我们建议直接估计QMI的衍生物，而不估计QMI本身。我们表明，直接估计的QMI的衍生物是更准确的比衍生物的估计QMI。最后，我们开发了一个线性监督降维算法，有效地使用所提出的导数估计，并通过实验证明，所提出的方法是更强大的离群值比现有的方法。
Abstract A typical goal of linear-supervised dimension reduction is to find a low-dimensional subspace of the input space such that the projected input variables preserve maximal information about the output variables. The dependence-maximization approach solves the supervised dimension-reduction problem through maximizing a statistical dependence between projected input variables and output variables. A well-known statistical dependence measure is mutual information (MI), which is based on the Kullback-Leibler (KL) divergence. However, it is known that the KL divergence is sensitive to outliers. Quadratic MI (QMI) is a variant of MI based on the distance, which is more robust against outliers than the KL divergence, and a computationally efficient method to estimate QMI from data, least squares QMI (LSQMI), has been proposed recently. For these reasons, developing a supervised dimension-reduction method based on LSQMI seems promising. However, not QMI itself but the derivative of QMI is needed for subspace search in linear-supervised dimension reduction, and the derivative of an accurate QMI estimator is not necessarily a good estimator of the derivative of QMI. In this letter, we propose to directly estimate the derivative of QMI without estimating QMI itself. We show that the direct estimation of the derivative of QMI is more accurate than the derivative of the estimated QMI. Finally, we develop a linear-supervised dimension-reduction algorithm that efficiently uses the proposed derivative estimator and demonstrate through experiments that the proposed method is more robust against outliers than existing methods.