Statistical Aggregation in Massive Data Environments

海量数据环境下的统计聚合

• 批准号: 0906023
• 负责人: Nan Lin
• 金额: $ 11.99万
• 依托单位: Washington University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-07-15 至 2012-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0906023&HistoricalAwards=false
• 关键词: Statistical; Aggregation; Massive; Data; Environments

Enormous amount of data are now being generated in many areas. Direct applications of existing statistical methods do not satisfy the computational need for performing on-line analytical processing (OLAP) on such massive data. Computer scientists have developed a data warehouse environment called data cube to reduce computational cost by compressing subsets given by some partitioning variables. Analysis of any subset can then possibly be achieved by aggregating the compressed data, and the computational cost becomes low because of no need to access the raw data. For complicated analyses, it is challenging to find proper compression and aggregation schemes and to study the statistical property of the aggregated analysis. Similar issues exist in another massive data environment, data stream. Existing development in these areas either aims to achieve lossless analysis, which has achieved very limited successes only for simple calculations, or provide no theoretical evaluation for the analysis from aggregation. The purpose of this proposed research is to develop statistically sound compression and aggregation methods for advanced statistical analysis of data cubes and data streams, use the above compression-then-aggregation strategy to improve computational efficiency of some statistical analysis, and develop the associated asymptotic theory.Massive data sets are common nowadays, and many traditional statistical techniques become inapplicable due to high computational costs. In this proposal, the investigator will extend the current data cube techniques to support more complicated OLAP of massive data sets by studying the statistical properties of the desired analysis. This interdisciplinary project will result in significant contributions to data warehousing, OLAP technology, and statistical computing. It will bring great impacts to important applications in large-scale medical studies, national and homeland security, stream data mining, high-performance computing, and information technology. This project's findings will be broadly disseminated to the academic community and industry through scholarly publications and conferences. We will also use the new findings as new course materials in education and training of information analysts and university students.

现在，在许多领域都产生了大量的数据。现有的统计方法的直接应用不能满足对这样的海量数据进行联机分析处理（OLAP）的计算需求。计算机科学家开发了一种称为数据立方体的数据仓库环境，通过压缩由一些划分变量给出的子集来减少计算成本。任何子集的分析都可以通过聚合压缩数据来实现，并且由于不需要访问原始数据，计算成本变得很低。对于复杂的分析，寻找合适的压缩和聚集方案以及研究聚集分析的统计特性是一个挑战。类似的问题也存在于另一个大规模数据环境中，即数据流。这些领域的现有发展要么旨在实现无损分析，仅在简单计算方面取得了非常有限的成功，要么没有为聚集分析提供理论评估。本研究的目的是开发统计上可靠的数据立方体和数据流的高级统计分析的压缩和聚合方法，使用上述压缩-聚合策略来提高某些统计分析的计算效率，并发展相关的渐近理论。在这个建议中，研究者将扩展目前的数据立方体技术，以支持更复杂的OLAP的海量数据集，通过研究所需的分析的统计特性。这个跨学科的项目将对数据仓库、OLAP技术和统计计算做出重大贡献。它将对大规模医学研究、国家和国土安全、流数据挖掘、高性能计算和信息技术等重要应用产生重大影响。该项目的研究结果将通过学术出版物和会议向学术界和工业界广泛传播。我们还将使用新发现作为信息分析师和大学生教育和培训的新课程材料。