Optimizing Near-Data Processing for Spark
Optimizing Near-Data Processing for Spark
复制标题
DOI：
10.1109/icdcs54860.2022.00067
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Sri Pramodh Rachuri;Arun Gantasala;Prajeeth Emanuel;Anshul Gandhi;Robert Foley;Peter Puhov;Theo Gkountouvas;H. Lei
中科院分区：
文献类型：
--
作者：
Sri Pramodh Rachuri;Arun Gantasala;Prajeeth Emanuel;Anshul Gandhi;Robert Foley;Peter Puhov;Theo Gkountouvas;H. Lei
资源分解（RD）是用于数据中心计算的新兴范例，其中采用资源优化的服务器来最小化资源碎片并提高资源利用率。在RD范式下部署的Apache Spark使用一组计算优化的服务器来运行执行器，并使用一组存储优化的服务器来托管HDFS上的数据。然而，从存储到计算集群的网络传输成为大数据处理的严重瓶颈。近数据处理（NDP）是一个概念，旨在通过将一些计算任务卸载（或“下推”）到存储集群来减轻这种情况下的网络负载。在RD模式下为Spark使用NDP是一项挑战，因为存储优化的服务器具有有限的计算资源，并且无法托管整个Spark处理堆栈。此外，即使这样一个轻量级的堆栈可以开发和部署在存储集群上，这是不完全明显的Spark查询将受益于下推，以及一个给定的查询的任务应该推到storage.This本文介绍了一个近数据处理系统的Spark，SparkNDP，旨在解决上述挑战的设计和实现。SparkNDP的工作原理是使用轻量级的SQL操作符库在存储集群上实现新的NDP Spark功能，然后开发一个分析模型，以帮助确定哪些Spark任务应该根据当前的网络和系统状态下推到存储。模拟和原型实现结果表明，与不将任何任务推送到存储的默认方法和将所有任务推送到存储的直接NDP方法相比，SparkNDP可以帮助减少Spark查询执行时间。
Resource disaggregation (RD) is an emerging paradigm for data center computing whereby resource-optimized servers are employed to minimize resource fragmentation and improve resource utilization. Apache Spark deployed under the RD paradigm employs a cluster of compute-optimized servers to run executors and a cluster of storage-optimized servers to host the data on HDFS. However, the network transfer from storage to compute cluster becomes a severe bottleneck for big data processing. Near-data processing (NDP) is a concept that aims to alleviate network load in such cases by offloading (or "pushing down") some of the compute tasks to the storage cluster. Employing NDP for Spark under the RD paradigm is challenging because storage-optimized servers have limited computational resources and cannot host the entire Spark processing stack. Further, even if such a lightweight stack could be developed and deployed on the storage cluster, it is not entirely obvious which Spark queries would benefit from pushdown, and which tasks of a given query should be pushed down to storage.This paper presents the design and implementation of a near-data processing system for Spark, SparkNDP, that aims to address the aforementioned challenges. SparkNDP works by implementing novel NDP Spark capabilities on the storage cluster using a lightweight library of SQL operators and then developing an analytical model to help determine which Spark tasks should be pushed down to storage based on the current network and system state. Simulation and prototype implementation results show that SparkNDP can help reduce Spark query execution times when compared to both the default approach of not pushing down any tasks to storage and the outright NDP approach of pushing all tasks to storage.