Outlier Detection in High-Dimensional Big Data using Bio-Inspired Methods for Emerging Applications in Engineering, Healthcare, and Business

在这个研究项目中，我将利用生物启发的方法来探索、设计和分析高维大数据中的离群点检测创新算法，并将新方法应用于新兴的工程应用(即计算机网络中的入侵检测)、商业(即企业财务报表中的欺诈检测)和医疗保健(即检测患者生命信号的异常)。 大数据的特征是体积、速度、多样性和准确性的四个(有时甚至更多)V，它被定义为数据集的集合，这些数据集如此庞大、动态和复杂，以至于使用传统的数据分析技术变得难以处理。在高维空间中，点之间的距离变得相对一致，数据点的最近邻居的概念变得没有意义。高维数据也有许多子空间的排列，这实际上是不可能全部检查的。处理如此大规模的多维数据在计算上复杂且代价高昂。 检测大数据中的离群点(对象与大多数数据有很大的不同和不一致)，特别是在高维数据和存在噪声的情况下，是一个重要的研究问题，由于它带来的科学挑战以及它支持的广泛现实世界应用，包括在工程、医疗保健、商业、环境和公共安全中，已经引起了研究界的广泛关注。 在这个研究项目中，我将探索在分布式平台上运行的降维、数据汇总和特征转换的新技术，结合模型集成来快速准确地检测离群点。我将探索受生物启发的算法，以搜索具有适应度函数的大排列空间，以最小化所选子空间中样本的稀疏性。 大数据分析依赖于可扩展的分布式平台，如Hadoop(支持MapReduce结构，用于并行分析大数据)和Spark(用于大规模数据处理的快速内存引擎)。在我的知识发现和数据挖掘实验室中，我们尝试了使用Hadoop和Spark并行处理任务。在这些经验的基础上，我们将在分布式平台上设计和实施我们的新型解决方案。 在这项研究计划中发现的解决方案和算法将应用于工程(分析互联网流量产生的大量高维数据，以检测网络入侵)、商业(在彭博社和CompuStat提供的4000多家公司的真实数据集中检测金融欺诈活动)和医疗保健(分析从患者收集的重要信号，包括体温、心跳、血压和心电信号，以检测异常)的新兴应用。

In this research program, I will explore, design, and analyze innovative algorithms for outlier detection in high-dimensional big data using bio-inspired approaches, and apply the new methods to emerging applications in engineering (namely, Intrusion detection in computer networks), business (namely, fraud detection in corporate financial statements), and healthcare (namely, detecting abnormalities in patient's vital signals). Big data, characterized by four (and sometimes more) V's of Volume, Velocity, Variety, and Veracity, is defined as a collection of data sets so large, dynamic, and complex that it becomes difficult to process using traditional data analytics techniques. In high dimensional spaces, distances between points become relatively uniform, and the notion of the nearest neighbors of a data point becomes meaningless. A high dimensional data has also numerous permutations of sub-spaces which are practically infeasible to be examined all. Processing such large-scale multi-dimensional data is computationally complex and expensive. Detecting outliers (objects considerably dissimilar and inconsistent with respect to the majority of data) in Big data, especially in high-dimensional data and in the presence of noise, is an important research problem which has drawn many attentions in research community due to scientific challenges it introduces, and a wide range of real-world applications it supports including in engineering, healthcare, business, environment, and public security. In this research program, I will explore novel techniques for dimension reduction, data summarization, and feature transformation running on distributed platforms, combined with ensemble of models to make fast and accurate detection of outliers. I will explore bio-inspired algorithms to search a large space of permutations with fitness functions minimizing sparsity of the samples in selected sub-spaces. Analysis of Big data relies on scalable distributed platforms such as Hadoop (which supports MapReduce structure for analysis of large data in parallel), and Spark (a fast in-memory engine for large scale data processing.). In my Knowledge Discovery and Data Mining Lab, we have experimented with processing tasks in parallel using Hadoop and Spark. Building on these experiences, we will design and implement our novel solutions on distributed platforms. The solutions and algorithms discovered in this research program will be applied to emerging applications in 3 areas of engineering (analyzing large volume of high-dimensional data generated by Internet traffic to detect intrusion in the network), business (detecting financial fraudulent activities in a real dataset of more than 4000 firms provided by Bloomberg, and CompuStat), and healthcare (analyzing vital signals collected from patients including temperature, heartbeat, blood pressure, and ECG signals to detect anomalies).