BIGDATA: Mid-Scale: DCM: A Formal Foundation for Big Data Management

分析大规模数据集的能力已经成为工业和科学中的重要工具，最近出现了许多系统来支持它。然而，深度数据分析的有效方法目前是高度接触的过程：它们需要一名高度专业化的专家，他彻底了解应用领域和相关的不同数据源，并且需要重复执行一系列数据探索、操作和转换步骤，以便为查询、机器学习或数据挖掘算法准备数据。该项目探索大数据管理的基础，最终目标是通过加快数据探索的瓶颈步骤，显著提高大数据分析的生产率。该项目集成了两个推动力：一个是理论研究，它导致了关于现代大规模系统中各种新的(即席)数据转换的复杂性的新的基本结果；另一个是系统研究，它导致了一个用于表达和优化即席数据分析技术的多平台软件中间件。该中间件旨在增强和集成现有的分析解决方案，以促进和改进社区感兴趣的方法，并与许多现有平台兼容。该项目的结果将使领域专家更容易在大数据和大型计算机集群上进行复杂的数据分析。所有研究结果将在现有大数据系统之上的中间件包中发布。该中间件包括项目期间开发的所有新算法、优化技术、容错和倾斜缓解机制以及通用聚合。此外，该项目还开发了一个基于Web的查询即服务接口，并将其部署到新的中间件。项目网站(http://myriadb.cs.washington.edu))提供对软件、其他结果和信息的访问。项目成果将纳入大数据分析方面的教育和推广活动，包括本科生、研究生和专业人员的新课程。

The ability to analyze massive-scale datasets has become an important tool both in industry and in the sciences and many systems have recently emerged to support it. However, effective methods for deep data analytics are currently high-touch processes: they require a highly specialized expert who thoroughly understands the application domain and pertinent disparate data sources and who needs to perform repeatedly a series of data exploration, manipulation and transformation steps to prepare the data for querying, machine learning or data mining algorithms. This project explores the foundations of big data management with the ultimate goal of significantly improving the productivity in big data analytics by accelerating the bottleneck step of data exploration. The project integrates two thrusts: a theoretical study, which leads to new fundamental results regarding the complexity of various new (ad hoc) data transformations in modern massive-scale systems, and a systems study, which leads to a multi-platform software middleware for expressing and optimizing ad hoc data analytics techniques. The middleware is designed to augment and integrate existing analytics solutions in order to facilitate and improve methods of interest to the community and compatible with many existing platforms.The results of this project will make it easier for domain experts to conduct complex data analysis on big data and on large computer clusters. All research results will be released in a middleware package layered on top of existing big-data systems. The middleware includes all the new algorithms, optimization techniques, fault-tolerance and skew mitigation mechanisms, and generalized aggregates developed during the project. In addition, the project develops and deploys a Web-based query-as-a-service interface to the new middleware. The project Web site (http://myriadb.cs.washington.edu) provides access to the software, additional results and information. Project results will be included in educational and outreach activities in big data analytics, including new curricula at the undergraduate, graduate, and professional levels.