BIGDATA: Small: DA: A Random Projection Approach

随着互联网的到来，网络流量、搜索和数据库环境中的许多应用程序都面临着非常大的、固有的高维或自然流传输的数据集。为了有效地解决这些超大规模的实际问题(如从海量数据中建立统计模型、实时网络流量监测和异常检测)，基于统计和概率的方法变得越来越流行。该方案旨在开发基于随机投影的海量数据统计方法，包括数据流算法、量化投影算法和稀疏投影算法。网络流量就是一个典型的例子。利用较小的存储空间实时有效地测量(和更新)网络流量对于检测异常事件至关重要，例如DDoS(分布式拒绝服务)攻击。对于许多应用，如数据库和机器学习，随机投影的适当量化将大大提高精度(就每比特方差而言)，并提供有效的索引和降维，以促进高效的搜索和学习。这项拟议的研究将解决随机预测开发中的一系列数学挑战问题。大量的统计学习和数值线性代数算法将被重新设计，以利用最先进的投影方法。如今，许多行业，如搜索，迫切需要能够有效处理海量数据的统计算法。预计将在该提案中开发的算法将与并行平台集成，以解决真正大规模的现实世界问题。研究成果将通过出版物、会议演示、行业访问和合作、教程和开放源码分发向从业人员传播。许多拟议的研究问题涉及统计分析，可能会继续有助于吸引统计学家/数学家从事大数据领域的工作。拟议的研究活动将通过创新的课程和研究培训，吸引本科生和研究生学习统计和工程学。

With the advent of Internet, numerous applications in the context of network traffic, search, and databases are faced with very large, inherently high-dimensional, or naturally streaming datasets. To effectively tackle these extremely large-scale practical problems (e.g., building statistical models from massive data, real-time network traffic monitoring and anomaly detection), methods based on statistics and probability have become increasingly popular. This proposal aims at developing theoretical, well-grounded statistical methods for massive data based on random projections, including data stream algorithms, quantized projection algorithms, and sparse projection algorithms.Massive data are often generated as high-rate streams. Network traffic is a typical example. Effective measurements (and updates) of network traffic in real-time using small storage space are crucial for detecting anomaly events, for example the DDoS (Distributed Denial of Service) attacks. For many applications such as databases and machine learning, appropriate quantization of random projections will substantially improve the accuracies (in terms of variance per bit) and provide efficient indexing and dimension reductions to facilitate efficient search and learning. The proposed research will tackle a series of mathematically challenging problems in the development of random projections. A wide range of statistical learning and numerical linear algebra algorithms will be re-engineered to take advantage of the state-the-art projection methods.These days, many industries such as search are in urgent demand for statistical algorithms which can effectively handle massive data. It is expected that algorithms to be developed in this proposal will be integrated with parallel platforms, to solve truly large-scale real-world problems. Research results will be disseminated to practitioners through publications, conference presentations, industry visits and collaborations, tutorials, and open-source distributions. Many of the proposed research problems involve statistical analysis and may continue to help attract statisticians/mathematicians to work on area of big data. The proposed research activities will engage both undergraduate and graduate students in statistics and engineering, through innovative curriculum and research training.