Understanding Behavior Trends of Big Data Frameworks in Ongoing Software-Defined Cyber-Infrastructure
Understanding Behavior Trends of Big Data Frameworks in Ongoing Software-Defined Cyber-Infrastructure
复制标题
了解正在进行的软件定义网络基础设施中大数据框架的行为趋势
DOI：
10.1145/3148055.3148079
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Rodero, Ivan
中科院分区：
文献类型：
--
作者：
Chen, Shouwei;Rodero, Ivan
随着数据分析应用在广泛的领域变得越来越重要，开发大规模和可持续的平台和软件基础设施来支持这些应用的能力，在推动科学和商业领域的研究和创新方面具有巨大的潜力。本文描述了大数据分析的两个主要框架（即Apache Hadoop和Spark）在一系列代表性应用程序中的性能和功能相关的行为趋势和权衡。它还评估了系统设计，如存储和网络技术以及功率上限技术。经验执行的实验结果为通过模拟探索大数据处理系统的软件定义基础设施的潜力提供了有意义的数据点。结果提供了对设计空间的更好理解，以构建多标准的以应用程序为中心的模型，并显示了软件定义的基础设施在执行时间、精力和成本方面的显著优势。它激发了对具有更深内存层次和软件定义基础设施的系统的内存处理公式的进一步研究。
As data analytics applications become increasingly important in a wide range of domains, the ability to develop large-scale and sustainable platforms and software infrastructure to support these applications has significant potential to drive research and innovation in both science and business domains. This paper characterizes performance and power-related behavior trends and tradeoffs of the two predominant frameworks for Big Data analytics (i.e., Apache Hadoop and Spark) for a range of representative applications. It also evaluates system design knobs, such as storage and network technologies and power capping techniques. Experimental results from empirical executions provide meaningful data points for exploring the potential of software-defined infrastructure for Big Data processing systems through simulation. The results provide better understanding of the design space to build multi-criteria application-centric models as well as show significant advantages of software-defined infrastructure in terms of execution time, energy and cost. It motivates further research focused on in-memory processing formulations regarding systems with deeper memory hierarchies and software-defined infrastructure.