Outlier Detection in High-Dimensional Big Data using Bio-Inspired Methods for Emerging Applications in Engineering, Healthcare, and Business

***在这个研究项目中，我将探索、设计和分析利用生物启发的方法在高维大数据中进行离群值检测的创新算法，并将这些新方法应用于工程（即计算机网络中的入侵检测）、商业（即企业财务报表中的欺诈检测）和医疗保健（即检测患者生命信号中的异常）等新兴应用。*******大数据的特点是体积（Volume）、速度（Velocity）、多样性（Variety）和准确性（Veracity）四个（有时更多）V，它被定义为大量、动态和复杂的数据集，使用传统的数据分析技术很难处理。在高维空间中，点之间的距离变得相对均匀，数据点的最近邻居的概念变得毫无意义。一个高维数据也有许多子空间的排列，这些排列实际上是不可能全部检验的。处理如此大规模的多维数据在计算上既复杂又昂贵。*******在大数据中，特别是在高维数据和存在噪声的情况下，检测异常值（与大多数数据相当不同和不一致的对象）是一个重要的研究问题，由于它带来的科学挑战，以及它支持的广泛的现实世界应用，包括工程、医疗保健、商业、环境和公共安全，已经引起了研究界的广泛关注。*******在这个研究项目中，我将探索在分布式平台上运行的降维、数据汇总和特征转换的新技术，并结合模型集成来快速准确地检测异常值。我将探索生物启发的算法，用适应度函数在选定的子空间中最小化样本的稀疏性来搜索一个大的排列空间。*******大数据分析依赖于可扩展的分布式平台，如Hadoop（支持MapReduce结构，用于并行分析大数据）和Spark（用于大规模数据处理的快速内存引擎）。在我的知识发现和数据挖掘实验室中，我们尝试使用Hadoop和Spark并行处理任务。基于这些经验，我们将在分布式平台上设计和实施我们的新解决方案。******本研究项目中发现的解决方案和算法将应用于3个领域的新兴应用：工程（分析互联网流量产生的大量高维数据，以检测网络入侵）、商业（在彭博社和CompuStat提供的4000多家公司的真实数据集中检测金融欺诈活动）和医疗保健(分析从患者收集的重要信号，包括体温、心跳、血压，以及心电图信号来检测异常)

***In this research program, I will explore, design, and analyze innovative algorithms for outlier detection in high-dimensional big data using bio-inspired approaches, and apply the new methods to emerging applications in engineering (namely, Intrusion detection in computer networks), business (namely, fraud detection in corporate financial statements), and healthcare (namely, detecting abnormalities in patient's vital signals).*******Big data, characterized by four (and sometimes more) V's of Volume, Velocity, Variety, and Veracity, is defined as a collection of data sets so large, dynamic, and complex that it becomes difficult to process using traditional data analytics techniques. In high dimensional spaces, distances between points become relatively uniform, and the notion of the nearest neighbors of a data point becomes meaningless. A high dimensional data has also numerous permutations of sub-spaces which are practically infeasible to be examined all. Processing such large-scale multi-dimensional data is computationally complex and expensive.*******Detecting outliers (objects considerably dissimilar and inconsistent with respect to the majority of data) in Big data, especially in high-dimensional data and in the presence of noise, is an important research problem which has drawn many attentions in research community due to scientific challenges it introduces, and a wide range of real-world applications it supports including in engineering, healthcare, business, environment, and public security.*******In this research program, I will explore novel techniques for dimension reduction, data summarization, and feature transformation running on distributed platforms, combined with ensemble of models to make fast and accurate detection of outliers. I will explore bio-inspired algorithms to search a large space of permutations with fitness functions minimizing sparsity of the samples in selected sub-spaces.*******Analysis of Big data relies on scalable distributed platforms such as Hadoop (which supports MapReduce structure for analysis of large data in parallel), and Spark (a fast in-memory engine for large scale data processing.). In my Knowledge Discovery and Data Mining Lab, we have experimented with processing tasks in parallel using Hadoop and Spark. Building on these experiences, we will design and implement our novel solutions on distributed platforms.******The solutions and algorithms discovered in this research program will be applied to emerging applications in 3 areas of engineering (analyzing large volume of high-dimensional data generated by Internet traffic to detect intrusion in the network), business (detecting financial fraudulent activities in a real dataset of more than 4000 firms provided by Bloomberg, and CompuStat), and healthcare (analyzing vital signals collected from patients including temperature, heartbeat, blood pressure, and ECG signals to detect anomalies).**