BIGDATA: Small: DA: A Random Projection Approach

随着Internet的出现，网络流量、搜索和数据库环境中的许多应用程序都面临着非常大的、固有的高维数据集或自然流数据集。为了有效地解决这些极其大规模的实际问题（例如，从海量数据中构建统计模型、实时网络流量监控和异常检测），基于统计和概率的方法越来越受欢迎。本提案旨在发展基于随机投影的海量数据的理论、有根据的统计方法，包括数据流算法、量化投影算法和稀疏投影算法。海量数据通常以高速率流的形式生成。网络流量就是一个典型的例子。使用小存储空间实时有效地测量（和更新）网络流量对于检测异常事件（例如DDoS（分布式拒绝服务）攻击）至关重要。对于许多应用程序，如数据库和机器学习，随机预测的适当量化将大大提高准确性（按每比特方差计算），并提供有效的索引和降维，以促进有效的搜索和学习。拟议的研究将解决随机预测发展过程中一系列具有数学挑战性的问题。广泛的统计学习和数值线性代数算法将被重新设计，以利用最先进的投影方法。如今，搜索等许多行业都迫切需要能够有效处理海量数据的统计算法。预计在本提案中开发的算法将与并行平台集成，以解决真正大规模的现实问题。研究成果将通过出版物、会议演讲、行业访问和合作、教程和开源发行等方式传播给从业者。许多提出的研究问题涉及统计分析，并可能继续帮助吸引统计学家/数学家从事大数据领域的工作。拟议的研究活动将通过创新课程和研究培训，吸引统计学和工程学的本科生和研究生。

With the advent of Internet, numerous applications in the context of network traffic, search, and databases are faced with very large, inherently high-dimensional, or naturally streaming datasets. To effectively tackle these extremely large-scale practical problems (e.g., building statistical models from massive data, real-time network traffic monitoring and anomaly detection), methods based on statistics and probability have become increasingly popular. This proposal aims at developing theoretical, well-grounded statistical methods for massive data based on random projections, including data stream algorithms, quantized projection algorithms, and sparse projection algorithms.Massive data are often generated as high-rate streams. Network traffic is a typical example. Effective measurements (and updates) of network traffic in real-time using small storage space are crucial for detecting anomaly events, for example the DDoS (Distributed Denial of Service) attacks. For many applications such as databases and machine learning, appropriate quantization of random projections will substantially improve the accuracies (in terms of variance per bit) and provide efficient indexing and dimension reductions to facilitate efficient search and learning. The proposed research will tackle a series of mathematically challenging problems in the development of random projections. A wide range of statistical learning and numerical linear algebra algorithms will be re-engineered to take advantage of the state-the-art projection methods.These days, many industries such as search are in urgent demand for statistical algorithms which can effectively handle massive data. It is expected that algorithms to be developed in this proposal will be integrated with parallel platforms, to solve truly large-scale real-world problems. Research results will be disseminated to practitioners through publications, conference presentations, industry visits and collaborations, tutorials, and open-source distributions. Many of the proposed research problems involve statistical analysis and may continue to help attract statisticians/mathematicians to work on area of big data. The proposed research activities will engage both undergraduate and graduate students in statistics and engineering, through innovative curriculum and research training.