SBIR Phase I: Data Squashing for Massive Data Analysis

这个小企业创新研究第一阶段项目的重点是包含数百万甚至数十亿个数据点的海量数据集。对这种规模的数据进行统计分析提出了新的计算挑战。压缩算法将大量数据集压缩成更小的数据集，以便在较小的（压缩的）数据集上执行的统计分析的输出可以复制在原始数据集上执行的相同统计分析的输出。压缩代表了采样的一种替代方法，作为处理大量数据的一种方式，其目的是在预测和推理准确性方面显著优于采样。压缩提供了几个优势：(1)计算密集型的统计过程，如非线性建模或大规模变量选择，当直接应用于大规模数据集时是不可行的，当应用于压缩表示时变得可行；(2)由于压缩后的数据集可能比原始的海量数据集小几个数量级，因此电子数据传播变得更加简单；(3)由于压缩数据集是合成的（不包含实际数据点），因此不存在披露风险。本研究的目的是批判性地评估压缩，并根据令人满意的评估，开发商业数据压缩软件产品。例如，在计算生物学、医疗监测系统、电信和天文学中，可能会遇到如此庞大的数据集。

This Small Business Innovation Research Phase I project focuses on massive datasets containing millions or even billions of data points. Statistical analyses of data on this scale present new computational challenges. Squashing algorithms compress massive datasets into much smaller ones so that outputs from statistical analyses carried out on the smaller (squashed) datasets reproduce outputs from the same statistical analyses carried out on the original datasets. Squashing represents an alternative to sampling as a way of dealing with massive data and aims to significantly outperform sampling in terms of predictive and inferential accuracy. Squashing affords several advantages: (1) Computationally intensive statistical procedures such as non-linear modeling or large-scale variable selection, infeasible when directly applied to the massive dataset, become feasible when applied to the squashed representation; (2) Since the squashed dataset may be several orders of magnitude smaller than the original massive dataset, electronic data dissemination becomes much simpler; (3) Because squashed datasets are synthetic (they contain no actual data points), they pose no disclosure risk. The objective of this research is to critically evaluate squashing, and, contingent on a satisfactory evaluation, develop a commercial data squashing software product. Such massive data sets can be encountered, for instance, in computational biology, medical surveillance systems, telecommunications, and astronomy.