CSR: Small: XooMR: Cross-Layer and Cross-Phase Cooperation for Fair and Efficient MapReduce

CSR：小：XooMR：跨层跨阶段合作实现公平高效的 MapReduce

• 批准号: 1564647
• 负责人: Weikuan Yu
• 金额: $ 37.81万
• 依托单位: Florida State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-08-16 至 2020-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1564647&HistoricalAwards=false
• 关键词: CSR; Small; XooMR; Cross; Layer

MapReduce has been widely leveraged as a new programming model to tap the power of parallel data processing for Big Data. More and more systems are being deployed to serve data-intensive analytics applications written in MapReduce. Many jobs can show up at the same time on a system with conflicting resource requirements. This project investigates cross-layer cooperation techniques to achieve system efficiency, cross-phase techniques to enhance job fairness and system throughput, and cross-job task co-scheduling techniques to exploit the temporal relationship among jobs for better throughput and services to analytic queries composed of multiple MapReduce jobs.This project has profound impacts in several aspects. These include (1) strengthening computer science courses at Auburn University, and enhancing instruction effectiveness with student research projects on MapReduce; (2) recruiting and cultivating students of diverse backgrounds, particularly under-represented minority and female student groups for careers in computing; (3) disseminating research results as publications, presentations, conference tutorials and demonstrations, releasing open-source software codes, and eventually pushing them for integration into the official Hadoop code base; and (4) collaborating with industry, strengthening research partnerships, and cultivating opportunities for technology transfer to industry.

MapReduce作为一种新的编程模型被广泛利用，以挖掘大数据并行数据处理的能力。越来越多的系统被部署到用MapReduce编写的数据密集型分析应用程序中。许多作业可能同时出现在资源需求冲突的系统上。该项目研究了实现系统效率的跨层协作技术，提高作业公平性和系统吞吐量的跨阶段技术，以及利用作业之间的时间关系提高吞吐量和服务于由多个MapReduce作业组成的分析查询的跨作业任务协同调度技术。这个项目在几个方面产生了深远的影响。这些措施包括：(1)加强奥本大学的计算机科学课程，并通过MapReduce学生研究项目提高教学效率；(2)招募和培养不同背景的学生，特别是少数族裔和女性学生群体从事计算机职业；(3)以出版物、演讲、会议教程、演示等形式传播研究成果，发布开源软件代码，最终推动其整合到Hadoop官方代码库中；(4)与产业界合作，加强研究伙伴关系，培育技术向产业界转移的机会。

项目成果

SHMEMGraph: Efficient and Balanced Graph Processing Using One-Sided Communication

• DOI: 10.1109/ccgrid.2018.00078
• 发表时间: 2018-05
• 期刊: 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)
• 作者: Huansong Fu;Manjunath Gorentla Venkata;Shaeke Salman;N. Imam;Weikuan Yu

Efficient User-Level Storage Disaggregation for Deep Learning

• DOI: 10.1109/cluster.2019.8891023
• 发表时间: 2019-09
• 期刊: 2019 IEEE International Conference on Cluster Computing (CLUSTER)
• 作者: Yue Zhu;Weikuan Yu;Bing Jiao;K. Mohror;A. Moody;Fahim Chowdhury

Compression of Time Evolutionary Image Data through Predictive Deep Neural Networks

通过预测深度神经网络压缩时间演化图像数据

• 发表时间: 2021
• 期刊: Cloud and Grid Computing
• 作者: Roy, Rupak;Sato, Kento;Bhattachrya, Subhadeep;Fang, Xingang;Joti, Yasumasa;Hatsui, Takaki;Hiraki, Toshiyuki;Guo, Jian;Yu, Weikuan.
• 通讯作者: Yu, Weikuan.

Multivariate modeling and two-level scheduling of analytic queries

• DOI: 10.1016/j.parco.2019.01.006
• 发表时间: 2019-07
• 期刊: Parallel Comput.
• 作者: Zhuo Liu;Amit Kumar Nath;Xiaoning Ding;Huansong Fu;Md. Muhib Khan;Weikuan Yu

Exploration of memory hybridization for RDD caching in Spark

• DOI: 10.1145/3315573.3329988
• 发表时间: 2019-06
• 期刊: Proceedings of the 2019 ACM SIGPLAN International Symposium on Memory Management
• 作者: Md. Muhib Khan;Muhammad Ahad Ul Alam;Amit Kumar Nath;Weikuan Yu