Performance optimization of parallel-distributed processing with checkpointing for cloud environment
Performance optimization of parallel-distributed processing with checkpointing for cloud environment
复制标题
DOI：
10.3934/jimo.2018014
复制
发表时间：
2017
影响因子：
1.3
通讯作者：
T. Hirai;H. Masuyama;S. Kasahara;Yutaka Takahashi
中科院分区：
文献类型：
--
作者：
T. Hirai;H. Masuyama;S. Kasahara;Yutaka Takahashi
在云计算中，最成功的应用程序框架是并行分布式处理，在这种框架中，巨大的任务被分成多个子任务，这些子任务在被称为工作者的机器集群上独立处理。由于其庞大的系统规模，云环境中Worker故障时有发生，子任务故障子任务导致任务处理延迟较大。一种缓解故障影响的方案是检查点方法，它将子任务的进度记录为检查点，并由其他工作者从最新的检查点恢复失败的子任务。这种方法可以减少任务的处理延迟。但是，频繁设置检查点是系统开销，因此必须正确设置检查点间隔。在本文中，我们考虑最小化任务处理时间的最优检查点数目。我们建立了一个具有检查点的并行分布式处理的随机模型，并近似地推导出平均任务处理时间和最优检查点数目的显式表达式。数值实验表明，在典型的云计算环境下，所提出的近似方法具有足够的精度。此外，所推导的最优检查点数目优于先前关于最小化并行分布式处理的任务处理时间的研究结果。
In cloud computing, the most successful application framework is parallel-distributed processing, in which an enormous task is split into a number of subtasks and those are processed independently on a cluster of machines referred to as workers. Due to its huge system scale, worker failures occur frequently in cloud environment and failed subtasks cause a large processing delay of the task. One of schemes to alleviate the impact of failures is checkpointing method, with which the progress of a subtask is recorded as checkpoint and the failed subtask is resumed by other worker from the latest checkpoint. This method can reduce the processing delay of the task. However, frequent checkpointing is system overhead and hence the checkpoint interval must be set properly. In this paper, we consider the optimal number of checkpoints which minimizes the task-processing time. We construct a stochastic model of parallel-distributed processing with checkpointing and approximately derive explicit expressions for the mean task-processing time and the optimal number of checkpoints. Numerical experiments reveal that the proposed approximations are sufficiently accurate on typical environment of cloud computing. Furthermore, the derived optimal number of checkpoints outperforms the result of previous study for minimizing the task-processing time on parallel-distributed processing.