BIGDATA: Mid-Scale: DCM: A Formal Foundation for Big Data Management

BIGDATA：中型：DCM：大数据管理的正式基础

• 批准号: 1247469
• 负责人: Dan Suciu
• 金额: $ 296.67万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-01-01 至 2017-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1247469&HistoricalAwards=false
• 关键词: BIGDATA; Mid; Scale; DCM; Formal

The ability to analyze massive-scale datasets has become an important tool both in industry and in the sciences and many systems have recently emerged to support it. However, effective methods for deep data analytics are currently high-touch processes: they require a highly specialized expert who thoroughly understands the application domain and pertinent disparate data sources and who needs to perform repeatedly a series of data exploration, manipulation and transformation steps to prepare the data for querying, machine learning or data mining algorithms. This project explores the foundations of big data management with the ultimate goal of significantly improving the productivity in big data analytics by accelerating the bottleneck step of data exploration. The project integrates two thrusts: a theoretical study, which leads to new fundamental results regarding the complexity of various new (ad hoc) data transformations in modern massive-scale systems, and a systems study, which leads to a multi-platform software middleware for expressing and optimizing ad hoc data analytics techniques. The middleware is designed to augment and integrate existing analytics solutions in order to facilitate and improve methods of interest to the community and compatible with many existing platforms.The results of this project will make it easier for domain experts to conduct complex data analysis on big data and on large computer clusters. All research results will be released in a middleware package layered on top of existing big-data systems. The middleware includes all the new algorithms, optimization techniques, fault-tolerance and skew mitigation mechanisms, and generalized aggregates developed during the project. In addition, the project develops and deploys a Web-based query-as-a-service interface to the new middleware. The project Web site (http://myriadb.cs.washington.edu) provides access to the software, additional results and information. Project results will be included in educational and outreach activities in big data analytics, including new curricula at the undergraduate, graduate, and professional levels.

分析大规模数据集的能力已经成为工业和科学领域的重要工具，最近出现了许多系统来支持它。然而，深度数据分析的有效方法目前是高接触过程：它们需要高度专业化的专家，该专家彻底理解应用领域和相关的不同数据源并且需要重复地执行一系列数据探索，操作和转换步骤，为查询、机器学习或数据挖掘算法准备数据。该项目探索了大数据管理的基础，最终目标是通过加速数据探索的瓶颈步骤来显着提高大数据分析的生产力。该项目集成了两个方面：理论研究，导致了关于现代大规模系统中各种新（ad hoc）数据转换复杂性的新的基本结果，以及系统研究，导致了用于表达和优化ad hoc数据分析技术的多平台软件中间件。该中间件旨在增强和集成现有的分析解决方案，以促进和改进社区感兴趣的方法，并与许多现有平台兼容。该项目的结果将使领域专家更容易对大数据和大型计算机集群进行复杂的数据分析。所有的研究结果都将在现有大数据系统之上的中间件包中发布。中间件包括所有新的算法，优化技术，容错和倾斜缓解机制，以及在项目期间开发的广义聚合。此外，该项目还开发和部署了一个基于Web的查询即服务接口到新的中间件。该项目的网站（myriadb.cs.washington.edu）提供了软件、其他结果和信息。项目成果将被纳入大数据分析的教育和推广活动，包括本科生、研究生和专业水平的新课程。