swMR: A Framework for Accelerating MapReduce Applications on Sunway Taihulight
swMR: A Framework for Accelerating MapReduce Applications on Sunway Taihulight
复制标题
swMR：神威·太湖光上加速MapReduce应用的框架
DOI：
10.1109/tetc.2018.2881265
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Depei Qian
中科院分区：
文献类型：
--
作者：
Xiaogang Zhong;Mingzhen Li;Hailong Yang;Yi Liu;Depei Qian
MapReduce has already been an indispensable framework for the programmers to develop big data applications at scale without concerning the underlying complex system details. However, such an important framework is missing on the Sunway many-core processor that powers the world-leading supercomputer Sunway Taihulight. This paper fills the gap by implementing an efficient MapReduce framework on Sunway many-core processor which takes full advantage of the architecture features such as local device memory and register communication. Specifically, we propose three different schemes that adopt different methods to partition the computation of map and reduce across the MPE and CPEs of Sunway processor. Especially in CPE-accelerated Dynamic Partitioning Scheme (CDPS), the processing role of the CPEs within each CPE pair can be transformed dynamically between map and reduce, which is effective to improve the load balance during runtime. In addition, we adopt the local device memory as well as register communication on each CPE to improve the efficiency of data access and communication. The experiment results demonstrate our MapReduce framework based on CDPS achieves 36.9× performance speedup on average across representative benchmarks.
登录
查看更多内容
影响因子：
12
作者：
Xujuan Zhou;Yue Xu;Yuefeng Li;A. Jøsang;Clive Cox
通讯作者：
Xujuan Zhou;Yue Xu;Yuefeng Li;A. Jøsang;Clive Cox
DOI：
10.1016/j.future.2013.05.001
发表时间：
2014-07
期刊：
Future Gener. Comput. Syst.
影响因子：
--
作者：
Xuan-Yi Lin;Yeh-Ching Chung
通讯作者：
Xuan-Yi Lin;Yeh-Ching Chung
DOI：
10.1007/s10586-014-0400-1
发表时间：
2015-03
期刊：
Cluster Computing
影响因子：
--
作者：
Hai Jiang;Yi Chen;Zhi Qiao;Tien-Hsiung Weng;Kuan-Ching Li
通讯作者：
Hai Jiang;Yi Chen;Zhi Qiao;Tien-Hsiung Weng;Kuan-Ching Li
影响因子：
5.5
作者：
Yaoguang Wang;Weiming Lu;Renjie Lou;Baogang Wei
通讯作者：
Yaoguang Wang;Weiming Lu;Renjie Lou;Baogang Wei
众核系统上MapReduce应用的内存访问拥塞优化
DOI：
10.1093/comjnl/bxu136
发表时间：
2016-03
期刊：
The Computer Journal
影响因子：
--
作者：
Liang Li;Endog Wang;xiaoshe Dong;Zhengdong Zhu
通讯作者：
Zhengdong Zhu