Scalable robust covariance and correlation estimates for data mining
Scalable robust covariance and correlation estimates for data mining
复制标题
用于数据挖掘的可扩展稳健协方差和相关性估计
DOI：
--
复制
发表时间：
2002
期刊：
影响因子：
--
通讯作者：
R. Zamar
中科院分区：
文献类型：
--
作者：
Fatemah A. Alqallaf;Kjell P. Konis;R. Douglas Martin;R. Zamar
协方差和相关估计在数据挖掘中有着重要的应用。在存在异常值的情况下，协方差和相关矩阵的经典估计是不可靠的。一小部分异常值，在某些情况下，甚至是一个单一的异常值，可能会扭曲经典的协方差和相关性估计，使它们实际上毫无用处。也就是说，绝大多数数据的相关性可能会被非常错误地报告；主成分变换可能具有误导性；通过马氏距离进行多维离群值检测可能无法检测到离群值。有大量关于稳健协方差和相关矩阵估计的统计文献，重点是具有高故障点和小最坏情况偏差的仿射等变估计。所有这些估计量在变量数目上都具有不可接受的指数复杂性，在观测数目上具有二次复杂性。在这篇文章中，我们集中讨论了几种稳健协方差和相关矩阵估计的变种，它们的变量数具有平方复杂性，观测数具有线性复杂性。这些估计器基于几种形式的成对稳健协方差和相关估计。所研究的估计器包括基于嵌入在最近由[14]提出的整体过程中的坐标方向稳健变换的两个快速估计器。我们证明了估计量具有吸引人的稳健性，并给出了一个在新的洞察Miner数据挖掘产品中使用其中一个估计量的例子。
Covariance and correlation estimates have important applications in data mining. In the presence of outliers, classical estimates of covariance and correlation matrices are not reliable. A small fraction of outliers, in some cases even a single outlier, can distort the classical covariance and correlation estimates making them virtually useless. That is, correlations for the vast majority of the data can be very erroneously reported; principal components transformations can be misleading; and multidimensional outlier detection via Mahalanobis distances can fail to detect outliers. There is plenty of statistical literature on robust covariance and correlation matrix estimates with an emphasis on affine-equivariant estimators that possess high breakdown points and small worst case biases. All such estimators have unacceptable exponential complexity in the number of variables and quadratic complexity in the number of observations. In this paper we focus on several variants of robust covariance and correlation matrix estimates with quadratic complexity in the number of variables and linear complexity in the number of observations. These estimators are based on several forms of pairwise robust covariance and correlation estimates. The estimators studied include two fast estimators based on coordinate-wise robust transformations embedded in an overall procedure recently proposed by [14]. We show that the estimators have attractive robustness properties, and give an example that uses one of the estimators in the new Insightful Miner data mining product.