Nonparametric Outlyingness and Descriptive Measures in Multivariate and General Data Settings

该项目扩展了核心统计科学两个密切互动领域的基础：非参数离群值识别和鲁棒描述性措施。强调了多变量和更复杂的数据类型。主体以外的数据点（“离群值”）可能会对统计分析产生不利影响，除非加以识别和考虑。这一关切是在新的情况下产生的，新的情况要求更新和扩大现有方法的提法。多变量建模与重尾数据和偏度和峰度的描述性措施，除了位置和偏度涉及增加关注离群值。多种新的数据类型（函数、形状、图像、集合、符号、传感器、流、树、图等）正在被复杂的，但特设的计算机科学数据挖掘方法需要更系统的处理。计算几何中的形状拟合问题带来了新形式的离群值问题。该研究开发了新的通用基础方法来检测离群值，消除了对仅处理离群值而不实际识别它们的算法的依赖，消除了对椭圆离群轮廓的过度依赖，并加强了对重尾数据的适应。项目的总体目标是为离群值检测建立扩展的概念统计基础，并为位置、离散度、偏度、峰度等的稳健描述性测量开发新的结构，随着计算资源的发展，统计数据分析和建模的范围正在扩大，以适应紧迫的新应用领域。所有科学和工程领域的数据都具有复杂的多维结构，通常具有大样本量，并涉及曲线，图像，文本和其他对象，通常处于流或网络结构中。这在检测和处理“异常”数据点（“离群值”）方面产生了重大的新问题。哪些案件不成立？“异常”病例如何影响对完整数据集的统计分析？当数据量很大并且涉及很多变量时，什么样的计算步骤可以有效地找到异常值？在欺诈检测、入侵检测、网络分析和数据挖掘等各种新情况下，哪些一般原则适用？如何定义“离群值”相对于融合的几个相关的数据集，例如图像，文本和传感器数据，可能会出现在国土安全？本研究报告探讨了这些基本的实际问题，目的是根据既定的统计原则制定新的方法。

This project extends foundations in two closely interactive areas of core statistical science: nonparametric outlier identification, and robust descriptive measures. Multivariate and more complex data types are emphasized. Data points apart from the main body ("outliers") can adversely affect statistical analyses unless identified and taken into account. This concern is arising in new contexts calling for updated and broadened formulations of current methods. Multivariate modeling with heavy tailed data and with skewness and kurtosis descriptive measures in addition to location and skewness involves increased concern with outliers. Diverse new data types (functional, shape, image, set, symbolic, sensor, stream, tree, graph, etc.) being treated by sophisticated but ad hoc computer science data mining approaches need more systematic treatment. Shape-fitting problems in computational geometry impose new forms of outlier issues. The study develops new general foundational approaches to outlier detection, eliminates reliance on algorithms that only handle outliers without actually identifying them in the input space, eliminates undue reliance upon elliptical outlyingness contours, and strengthens the accommodation of heavy tailed data. The overall project goals are to establish extended conceptual statistical foundations for outlier detection and to develop new structures for robust descriptive measures of location, dispersion, skewness, kurtosis, etc., with the aim of broad application across general data settings.With advancing computational resources, the scope of statistical data analysis and modeling is widening to accommodate pressing new arenas of application. Data in all areas of science and engineering has complex multidimensional structure, typically with large sample sizes and involving curves, images, text, and other objects, often within astream or network structure. This is generating major new problems in detection and handling of "anomalous" data points ("outliers"). Which cases stand apart? How do the "unusual" cases impact statistical analyses on the full data set? What computational steps efficiently find the outliers when the data is massive and involves many variables? What general principles apply across diverse new situations such as fraud detection, intrusion detection, network analysis, and data mining? How to define "outlier" relative to a fusion of several related data sets, for example image, text, and sensor data, as might arise in Homeland Security? This study addresses these basic practical questions with the aim of developing new methodological approaches soundly based upon established statistical principles.