Abstract cost models for distributed data-intensive computations
Abstract cost models for distributed data-intensive computations
复制标题
分布式数据密集型计算的抽象成本模型
DOI：
10.1007/s10619-018-7244-2
复制
发表时间：
2019
影响因子：
1.2
通讯作者：
Yao, Yi
中科院分区：
文献类型：
--
作者：
Li, Rundong;Mi, Ningfang;Riedewald, Mirek;Sun, Yizhou;Yao, Yi
我们考虑分布式架构上的数据分析工作负载，特别是商用机器集群。为了找到最小化运行时间的作业划分，需要一个成本模型，我们更准确地将其称为最大完工时间模型。在试图找到尽可能简单但又足够准确的此类模型时，我们探索了输入、输出和计算复杂性的分段线性函数。它们是抽象的，因为它们捕获了基本的算法属性，但不需要对系统和实现细节(如磁盘访问次数)进行显式建模。我们展示了如何利用简化的功能结构来降低优化成本。在一般情况下，我们确定了一个可用于搜索空间剪枝的下界。对于具有同类任务的应用程序，我们进一步演示了如何将该模型直接集成到最大完工时间优化过程中，从而将搜索空间的维度降低了几个数量级。实验结果证明了良好的预测质量和成功的跨各种算子和集群架构的最大完工时间优化。
We consider data analytics workloads on distributed architectures, in particular clusters of commodity machines. To find a job partitioning that minimizes running time, a cost model, which we more accurately refer to as makespan model, is needed. In attempting to find the simplest possible, but sufficiently accurate, such model, we explore piecewise linear functions of input, output, and computational complexity. They are abstract in the sense that they capture fundamental algorithm properties, but do not require explicit modeling of system and implementation details such as the number of disk accesses. We show how the simplified functional structure can be exploited to reduce optimization cost. In the general case, we identify a lower bound that can be used for search-space pruning. For applications with homogeneous tasks, we further demonstrate how to directly integrate the model into the makespan optimization process, reducing search-space dimensionality and thus complexity by orders of magnitude. Experimental results provide evidence of good prediction quality and successful makespan optimization across a variety of operators and cluster architectures.
DOI：
10.14778/2733004.2733005
发表时间：
2014-08
期刊：
Proc. VLDB Endow.
影响因子：
--
作者：
Juwei Shi;Jia Zou;Jiaheng Lu;Zhao Cao;Shiqiang Li;Chen Wang
通讯作者：
Juwei Shi;Jia Zou;Jiaheng Lu;Zhao Cao;Shiqiang Li;Chen Wang
禁止杀手机器人的理由
影响因子：
22.7
作者：
R. Arkin
通讯作者：
R. Arkin