BIGDATA: Small: DCM: DA: Building a Mergeable and Interactive Distributed Data Layer for Big Data Summarization Systems

今天的大数据以分布式方式存储在许多不同的机器或数据源中。这给对整个数据集进行有效分析带来了新的算法和系统挑战。为了解决这些困难，PI正在构建中间(可合并和交互的分布式数据层)摘要系统，并将其部署在大型真实世界数据集上。中间系统构建并维护一类特殊的摘要，可以高效地构建和更新这些摘要，同时仍然允许对繁重的尾部进行细粒度分析。可合并摘要可以表示具有大小和精度折衷保证的任何数据集，并且可以将任何两个这样的摘要合并以创建具有相同大小精度折衷的新摘要。交互式摘要可以快速适应指定的数据查询范围，同时保持相对于该范围内的数据的相同大小精度折衷。这允许准确、高效地分析大数据的小子集。中间系统使不同的大数据用户能够通过使用数据摘要开发一系列高效且可扩展的数据分析任务。中间系统正在领域专家的帮助下进行评估和改进。由于基于数据摘要的分析成为处理大数据的标准技术的一部分的前景诱人，这项研究对国家政府机构、研究机构、教育系统和高科技行业产生了更广泛的影响。我们的广泛影响还延伸到学术界和社区外展，通过设计和开发大数据课程和教育，以及通过简洁的摘要让普通公众参与理解和使用大数据。

Big data today is stored in a distributed fashion across many different machines or data sources. This poses new algorithmic and system challenges to performing efficient analysis on the full data set. To address these difficulties, the PIs are building the MIDDLE (Mergeable and Interactive Distributed Data LayEr) Summarization System and deploying it on large real-world datasets. The MIDDLE system builds and maintains a special class of summaries that can be efficiently constructed and updated while still allowing fine-grained analysis on the heavy tail. Mergeable summaries can represent any data set with a guaranteed tradeoff between size and accuracy, and any two such summaries can be merged to create a new summary with the same size-accuracy tradeoff.Interactive summaries can be quickly adapted to a specified query range of data while maintaining the same size-accuracy tradeoffs relative to the data in that range. This allows accurate efficient analysis to zero-in on small subsets of big data.The MIDDLE system enables different big data users to develop a wide spectrum of efficient and scalable data analytic tasks through the use of data summaries. The MIDDLE system is being evaluated and refined with the aid of domain experts. Since the prospect of data-summary-based analytics becoming a part of standard techniques in processing big data is tantalizing, this research generates broader impacts on the nation's government agencies, research institutes, education system, and high-tech industries. Our broad impacts also extend to academia and community outreach, through the design and development big data curriculum and education, and the involvement of general public in understanding and using big data through concise summaries.