FlexAnalytics: A Flexible Data Analytics Framework for Big Data Applications with I/O Performance Improvement
FlexAnalytics: A Flexible Data Analytics Framework for Big Data Applications with I/O Performance Improvement
复制标题
DOI：
10.1016/j.bdr.2014.07.001
复制
发表时间：
2014-08-01
影响因子：
3.3
通讯作者：
Chen, Hsuan-Wei Michelle
中科院分区：
文献类型：
--
作者：
Zou, Hongbo;Yu, Yongen;Chen, Hsuan-Wei Michelle
越来越大规模的应用程序正在生成前所未有的数据量。然而，高端计算机的计算能力和 I/O 能力之间日益扩大的差距给数据分析带来了严重的瓶颈。现场分析不是将数据从源移动到输出存储，而是在模拟运行时处理输出数据。然而，现场数据分析会因模拟而引发更多的计算资源争用。此类争论严重损害了 HPE 上的模拟性能。由于不同的数据处理策略对性能和成本有不同的影响，因此数据分析的位置需要具有灵活性。在本文中，我们探索并分析了 I/O 路径上的几种潜在的数据分析放置策略。为了找出在给定情况下减少数据移动的最佳策略，我们在本文中提出了一种灵活的数据分析（FlexAnalytics）框架。基于此框架，开发了用于分析放置的 FlexAnalytics 原型系统。 FlexAnalytics系统增强了HEC平台上当前I/O堆栈的可扩展性和灵活性，对于数据预处理、运行时数据分析和可视化以及大规模数据传输非常有用。研究中应用了科学数据压缩和远程可视化这两个用例来验证 FlexAnalytics 的性能。实验结果表明，FlexAnalytics框架增加了数据传输带宽并提高了应用程序端到端传输性能。 (C) 2014 Elsevier Inc. 保留所有权利。
Increasingly larger scale applications are generating an unprecedented amount of data. However, the increasing gap between computation and I/O capacity on High End Computing machines makes a severe bottleneck for data analysis. Instead of moving data from its source to the output storage, in-situ analytics processes output data while simulations are running. However, in-situ data analysis incurs much more computing resource contentions with simulations. Such contentions severely damage the performance of simulation on HPE. Since different data processing strategies have different impact on performance and cost, there is a consequent need for flexibility in the location of data analytics. In this paper, we explore and analyze several potential data-analytics placement strategies along the I/O path. To find out the best strategy to reduce data movement in given situation, we propose a flexible data analytics (FlexAnalytics) framework in this paper. Based on this framework, a FlexAnalytics prototype system is developed for analytics placement. FlexAnalytics system enhances the scalability and flexibility of current I/O stack on HEC platforms and is useful for data pre-processing, runtime data analysis and visualization, as well as for large-scale data transfer. Two use cases - scientific data compression and remote visualization -have been applied in the study to verify the performance of FlexAnalytics. Experimental results demonstrate that FlexAnalytics framework increases data transition bandwidth and improvesthe application end-toend transfer performance. (C) 2014 Elsevier Inc. Allrightsreserved.