Statistical Aggregation in Massive Data Environments

现在，许多领域都在产生海量数据。现有统计方法的直接应用不能满足对海量数据进行在线分析处理(OLAP)的计算需要。计算机科学家开发了一种名为数据立方体的数据仓库环境，通过压缩由一些划分变量给出的子集来降低计算成本。然后，可以通过聚集压缩数据来实现对任何子集的分析，并且由于不需要访问原始数据，计算成本变得很低。对于复杂的分析，寻找合适的压缩和聚集方案以及研究聚集分析的统计特性是具有挑战性的。类似的问题也存在于另一个海量数据环境-数据流中。这些领域的现有发展要么旨在实现无损分析，仅通过简单的计算取得非常有限的成功，要么没有为聚集分析提供理论评估。该研究的目的是为数据立方体和数据流的高级统计分析开发统计合理的压缩和聚集方法，使用上述压缩-然后-聚集策略来提高某些统计分析的计算效率，并发展相关的渐近理论。在该方案中，研究人员将扩展现有的数据立方体技术，通过研究期望分析的统计特性来支持更复杂的海量数据集的OLAP。这一跨学科项目将对数据仓库、OLAP技术和统计计算做出重大贡献。它将对大规模医学研究、国家和国土安全、流数据挖掘、高性能计算和信息技术等领域的重要应用产生重大影响。该项目的研究结果将通过学术出版物和会议广泛传播给学术界和产业界。我们还将使用新的调查结果作为教育和培训信息分析员和大学生的新课程材料。

Enormous amount of data are now being generated in many areas. Direct applications of existing statistical methods do not satisfy the computational need for performing on-line analytical processing (OLAP) on such massive data. Computer scientists have developed a data warehouse environment called data cube to reduce computational cost by compressing subsets given by some partitioning variables. Analysis of any subset can then possibly be achieved by aggregating the compressed data, and the computational cost becomes low because of no need to access the raw data. For complicated analyses, it is challenging to find proper compression and aggregation schemes and to study the statistical property of the aggregated analysis. Similar issues exist in another massive data environment, data stream. Existing development in these areas either aims to achieve lossless analysis, which has achieved very limited successes only for simple calculations, or provide no theoretical evaluation for the analysis from aggregation. The purpose of this proposed research is to develop statistically sound compression and aggregation methods for advanced statistical analysis of data cubes and data streams, use the above compression-then-aggregation strategy to improve computational efficiency of some statistical analysis, and develop the associated asymptotic theory.Massive data sets are common nowadays, and many traditional statistical techniques become inapplicable due to high computational costs. In this proposal, the investigator will extend the current data cube techniques to support more complicated OLAP of massive data sets by studying the statistical properties of the desired analysis. This interdisciplinary project will result in significant contributions to data warehousing, OLAP technology, and statistical computing. It will bring great impacts to important applications in large-scale medical studies, national and homeland security, stream data mining, high-performance computing, and information technology. This project's findings will be broadly disseminated to the academic community and industry through scholarly publications and conferences. We will also use the new findings as new course materials in education and training of information analysts and university students.