Optimized Rollback and Re-computation
Optimized Rollback and Re-computation
复制标题
优化回滚和重新计算
DOI：
10.1109/hicss.2013.434
复制
发表时间：
2013
期刊：
影响因子：
--
通讯作者：
Ashish Gehani
中科院分区：
文献类型：
--
作者：
Hasnain Lakhani;Rashid Tahir;Azeem Aqil;Fareed Zaffar;Dawood Tariq;Ashish Gehani
可以使用工作流管理系统来执行大数据处理任务。当管道中的输入数据或程序被修改时，必须重新执行工作流，以确保最终输出数据得到更新以反映这些变化。由于这种重新计算可能会消耗大量资源，因此优化系统以避免冗余计算是可取的。对于工作流，文件之间的依赖关系在一开始就被指定，并且可以利用这些关系来跟踪当特定文件发生变化时哪些程序需要重新执行。当不存在预定义的工作流时，当前的分布式系统无法提供这种功能。在本文中，我们提出了一种架构，该架构通过利用数据的溯源沿着隐式依赖图传播变化，从而提供产生正确输出以及快速重新执行的功能。我们通过对回滚和重新执行方案进行性能分析，探讨了存储和可用性之间的权衡。
Large data processing tasks can be effected using workflow management systems. When either the input data or the programs in the pipeline are modified, the workflow must be re-executed to ensure that the final output data is updated to reflect the changes. Since such re-computation can consume substantial resources, optimizing the system to avoid redundant computation is desirable. In the case of a workflow, the dependency relationships between files are specified at the outset and can be leveraged to track which programs need to be re-executed when particular files change. Current distributed systems cannot provide such functionality when no predefined workflows exist. In this paper, we present an architecture that provides functionality to produce both correct output as well as fast re-execution by leveraging the provenance of data to propagate changes along an implicit dependency graph. We explore the tradeoff between storage and availability by presenting a performance analysis of our rollback and re-execution scheme.