Characterizing I/O optimization opportunities for array-centric applications on HDFS

Characterizing I/O optimization opportunities for array-centric applications on HDFS
复制标题

DOI：
10.1109/hpec.2018.8547529
复制
发表时间：
2018-09
期刊：
2018 IEEE High Performance extreme Computing Conference (HPEC)
影响因子：
--
通讯作者：
Donghe Kang;Vedang Patel;Kalyan Khandrika;Spyros Blanas;Yang Wang;S. Parthasarathy
Donghe Kang;Vedang Patel;Kalyan Khandrika;Spyros Blanas;Yang Wang;S. Parthasarathy
中科院分区：
其他
文献类型：
--
作者：
Donghe Kang;Vedang Patel;Kalyan Khandrika;Spyros Blanas;Yang Wang;S. Parthasarathy

文献摘要

被引文献

相似文献

在日益复杂的以阵列为中心的分析和并行文件系统的基于字节流的POSIX接口之间存在阻抗不匹配。这种不匹配在数据密集型科学应用中尤为严重。本文研究了性能瓶颈，并描述了在计算天文学管道和Hadoop分布式文件系统（HDFS）的上下文中缓解这些瓶颈的优化方法。我们发现，快速的数据摄取和智能对象整合有望将I/O性能提高两个数量级。
An impedance mismatch exists between the increasing sophistication of array-centric analytics and the bytestream-based POSIX interface of parallel file systems. This mismatch is particularly acute in data-intensive scientific applications. This paper examines performance bottlenecks and describes optimizations to alleviate them in the context of computational astronomy pipelines and the Hadoop distributed file system (HDFS). We find that fast data ingestion and intelligent object consolidation promise to accelerate I/O performance by two orders of magnitude.