EAGER: Harnessing Dependency to Achieve Efficient Resource Management in Scalable Multi-stage Data Processing Systems

EARLY概念探索性研究（EAGER）赠款奖将为大型系统开发一个有效的资源管理框架，以服务于大量的多阶段数据处理工作。主要的研究目标包括实现跨作业的数据流，并利用阶段之间的依赖关系，以有效地分配系统资源。当前的数据处理框架在其执行期间封装每个作业，这禁用了作业（阶段）之间可能的数据共享和查询优化。这项研究的拟议工作将允许用户指定丰富的工作元数据，确定每个阶段的数据输入。处理服务器将根据作业元数据识别跨作业数据流。此外，一个重要的贡献将是一个依赖性驱动的资源管理计划，仔细审查独特的重叠过渡和连续阶段之间的依赖关系。拟议的工作将改善现有的多阶段数据处理平台。此外，与工业界联系人的合作具有快速技术转让的潜力。

This EArly-concept Grants for Exploratory Research (EAGER) award will develop an efficient resource management framework for a large-scale system to serve a large volume of multi-stage data processing jobs. The major research goals include enabling cross-job data flows, and exploiting the dependency between stages to efficiently allocate system resources. The current data processing framework encapsulates each job during its execution which disables possible data sharing between jobs (stages) and query optimization. The proposed work for this research will allow users to specify enriched job meta-data that identifies the data input of each stage. Processing servers will recognize the cross-job data flows according to the job meta-data. In addition, a significant contribution will be a dependency-driven resource management scheme that carefully examines the unique overlapping transition and dependency between consecutive stages. The proposed work will improve the existing platforms of multi-stage data processing. In addition, the collaborations with industrial contacts have the potential for rapid technology transfer.