Regularized covariance matrix estimation with high dimensional data for supervised anomaly detection problems
Regularized covariance matrix estimation with high dimensional data for supervised anomaly detection problems
复制标题
用于监督异常检测问题的高维数据正则协方差矩阵估计
DOI：
10.1109/ijcnn.2016.7727554
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
Kiran Byadarhaly
中科院分区：
文献类型：
--
作者：
D. Nikovski;Kiran Byadarhaly
我们解决的问题，估计高维协方差矩阵（CM）的监督异常检测的明确目的，在数据点的数量n低于其维数p的情况下，这是越来越常见的物联网的出现，使人们有可能同时从许多传感器收集数据，导致非常高维的数据点。当我们试图通过多变量高斯分布对系统的正常行为进行建模来对这样的数据进行异常检测时，并且n <p，样本CM是奇异的，并且在没有某种形式的正则化的情况下不能直接使用。与现有的CM正则化方法相比，这些方法旨在准确地拟合训练数据，我们提出了一种CM估计的正则化算法，其直接目的是最大化最终需要解决的决策问题（异常检测）的接收者操作者特征（AUROC）下的区域。测试问题的实验表明，该算法的能力，以发现CM估计显着优于现有的估计方法，不知道的决策任务，他们产生的CM将被用于异常检测。
We address the problem of estimating high-dimensional covariance matrices (CM) for the explicit purpose of supervised anomaly detection, in the case when the number n of data points is lower than their dimensionality p. This is increasingly common with the emergence of the Internet of Things that makes it possible to collect data from many sensors simultaneously, resulting in very high-dimensional data points. When we attempt to perform anomaly detection for such data by modeling the normal behavior of the system by means of a multivariate Gaussian distribution, and n <; p, the sample CM is singular, and cannot be used directly without some form of regularization. In contrast to existing methods for CM regularization that aim to fit the training data accurately, we propose a regularization algorithm for CM estimation that directly aims to maximize the area under the resulting receiver-operator characteristic (AUROC) for the ultimate decision problem that needs to be solved: anomaly detection. Experiments on test problems demonstrate the ability of the proposed algorithm to find CM estimates significantly better at anomaly detection than existing estimation methods that are unaware of the decision task that the CMs they produce will be used in.