BIGDATA: Small: DCM: DA: Building a Mergeable and Interactive Distributed Data Layer for Big Data Summarization Systems

今天的大数据以分布式的方式存储在许多不同的机器或数据源上。这对在完整数据集上执行有效分析提出了新的算法和系统挑战。为了解决这些困难，pi正在构建MIDDLE（可合并和交互式分布式数据层）总结系统，并将其部署在大型真实数据集上。MIDDLE系统构建并维护一类特殊的摘要，可以有效地构造和更新这些摘要，同时仍然允许对重尾进行细粒度分析。可合并摘要可以表示任何数据集，并保证在大小和精度之间进行权衡，并且可以合并任意两个这样的摘要以创建具有相同大小-精度权衡的新摘要。交互式摘要可以快速适应指定的数据查询范围，同时保持与该范围内的数据相同的大小-精度权衡。这允许对大数据的小子集进行准确有效的分析。MIDDLE系统使不同的大数据用户能够通过使用数据摘要开发广泛的高效和可扩展的数据分析任务。MIDDLE系统正在领域专家的帮助下进行评估和改进。由于基于数据摘要的分析成为处理大数据的标准技术的一部分的前景是诱人的，这项研究对国家政府机构、研究机构、教育系统和高科技产业产生了更广泛的影响。我们的广泛影响还延伸到学术界和社区外展，通过设计和开发大数据课程和教育，以及通过简明摘要让公众参与理解和使用大数据。

Big data today is stored in a distributed fashion across many different machines or data sources. This poses new algorithmic and system challenges to performing efficient analysis on the full data set. To address these difficulties, the PIs are building the MIDDLE (Mergeable and Interactive Distributed Data LayEr) Summarization System and deploying it on large real-world datasets. The MIDDLE system builds and maintains a special class of summaries that can be efficiently constructed and updated while still allowing fine-grained analysis on the heavy tail. Mergeable summaries can represent any data set with a guaranteed tradeoff between size and accuracy, and any two such summaries can be merged to create a new summary with the same size-accuracy tradeoff.Interactive summaries can be quickly adapted to a specified query range of data while maintaining the same size-accuracy tradeoffs relative to the data in that range. This allows accurate efficient analysis to zero-in on small subsets of big data.The MIDDLE system enables different big data users to develop a wide spectrum of efficient and scalable data analytic tasks through the use of data summaries. The MIDDLE system is being evaluated and refined with the aid of domain experts. Since the prospect of data-summary-based analytics becoming a part of standard techniques in processing big data is tantalizing, this research generates broader impacts on the nation's government agencies, research institutes, education system, and high-tech industries. Our broad impacts also extend to academia and community outreach, through the design and development big data curriculum and education, and the involvement of general public in understanding and using big data through concise summaries.