EAGER: Harnessing Dependency to Achieve Efficient Resource Management in Scalable Multi-stage Data Processing Systems

点击翻译按钮获取中文摘要

This EArly-concept Grants for Exploratory Research (EAGER) award will develop an efficient resource management framework for a large-scale system to serve a large volume of multi-stage data processing jobs. The major research goals include enabling cross-job data flows, and exploiting the dependency between stages to efficiently allocate system resources. The current data processing framework encapsulates each job during its execution which disables possible data sharing between jobs (stages) and query optimization. The proposed work for this research will allow users to specify enriched job meta-data that identifies the data input of each stage. Processing servers will recognize the cross-job data flows according to the job meta-data. In addition, a significant contribution will be a dependency-driven resource management scheme that carefully examines the unique overlapping transition and dependency between consecutive stages. The proposed work will improve the existing platforms of multi-stage data processing. In addition, the collaborations with industrial contacts have the potential for rapid technology transfer.