EAGER: Harnessing Dependency to Achieve Efficient Resource Management in Scalable Multi-stage Data Processing Systems

EAGER：利用依赖性在可扩展的多级数据处理系统中实现高效的资源管理

• 批准号: 1552525
• 负责人: Bo Sheng
• 金额: $ 12.5万
• 依托单位: University of Massachusetts Boston
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2017-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1552525&HistoricalAwards=false
• 关键词: EAGER; Harnessing; Dependency; Achieve; Efficient

This EArly-concept Grants for Exploratory Research (EAGER) award will develop an efficient resource management framework for a large-scale system to serve a large volume of multi-stage data processing jobs. The major research goals include enabling cross-job data flows, and exploiting the dependency between stages to efficiently allocate system resources. The current data processing framework encapsulates each job during its execution which disables possible data sharing between jobs (stages) and query optimization. The proposed work for this research will allow users to specify enriched job meta-data that identifies the data input of each stage. Processing servers will recognize the cross-job data flows according to the job meta-data. In addition, a significant contribution will be a dependency-driven resource management scheme that carefully examines the unique overlapping transition and dependency between consecutive stages. The proposed work will improve the existing platforms of multi-stage data processing. In addition, the collaborations with industrial contacts have the potential for rapid technology transfer.

这项早期概念探索性研究资助 (EAGER) 奖项将为大型系统开发一个高效的资源管理框架，为大量多阶段数据处理作业提供服务。主要研究目标包括实现跨作业数据流，以及利用阶段之间的依赖关系来有效地分配系统资源。当前的数据处理框架在执行期间封装了每个作业，这禁用了作业（阶段）之间可能的数据共享和查询优化。这项研究的拟议工作将允许用户指定丰富的工作元数据，以识别每个阶段的数据输入。处理服务器将根据作业元数据识别跨作业数据流。此外，一个重要的贡献将是依赖驱动的资源管理方案，该方案仔细检查连续阶段之间独特的重叠过渡和依赖关系。拟议的工作将改进现有的多阶段数据处理平台。此外，与工业界的合作具有快速技术转让的潜力。