KheOps: Cost-effective Repeatability, Reproducibility, and Replicability of Edge-to-Cloud Experiments
KheOps: Cost-effective Repeatability, Reproducibility, and Replicability of Edge-to-Cloud Experiments
复制标题
DOI：
10.1145/3589806.3600032
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Daniel Rosendo;K. Keahey;Alexandru Costan;Matthieu Simonin;P. Valduriez;Gabriel Antoniu
中科院分区：
文献类型：
--
作者：
Daniel Rosendo;K. Keahey;Alexandru Costan;Matthieu Simonin;P. Valduriez;Gabriel Antoniu
用于计算和分析的分布式基础设施现在正朝着互连的生态系统发展，允许复杂的科学工作流在从物联网边缘设备到云，有时甚至是超级计算机（计算连续体）的混合系统中执行。了解部署在如此复杂的边缘到云连续体上的大规模工作流的性能权衡是一项挑战。为了实现这一目标，需要系统地进行实验，以实现其可重复性，并允许其他研究人员在不同的基础设施上复制研究和获得的结论。这就分解为将大量实验性需求和约束与低级基础设施设计选择相协调的繁琐过程。为了解决分布式协作实验的主要最先进方法（如Google Colab，Kaggle和Code Ocean）的局限性，我们提出了KheOps，这是一个专门设计用于实现边缘到云实验的成本效益可重复性和可复制性的协作环境。KheOps由三个核心元素组成：（1）实验存储库;（2）笔记本环境;（3）多平台实验方法。我们用一个实际的边缘到云应用程序来说明KheOps。评估探讨了文章中描述的实验作者的观点（旨在使他们的实验可重复）和读者的观点（旨在复制实验）。结果显示了KheOps如何帮助作者在Grid5000 + FIT IoT LAB测试平台上系统地执行可重复和可再现的实验。此外，KheOps帮助读者在不同的基础设施（如Chameleon Cloud + CHI@Edge测试平台）中经济高效地复制作者的实验，并以高准确度获得相同的结论（所有性能指标> 88%）。
Distributed infrastructures for computation and analytics are now evolving towards an interconnected ecosystem allowing complex scientific workflows to be executed across hybrid systems spanning from IoT Edge devices to Clouds, and sometimes to supercomputers (the Computing Continuum). Understanding the performance trade-offs of large-scale workflows deployed on such complex Edge-to-Cloud Continuum is challenging. To achieve this, one needs to systematically perform experiments, to enable their reproducibility and allow other researchers to replicate the study and the obtained conclusions on different infrastructures. This breaks down to the tedious process of reconciling the numerous experimental requirements and constraints with low-level infrastructure design choices. To address the limitations of the main state-of-the-art approaches for distributed, collaborative experimentation, such as Google Colab, Kaggle, and Code Ocean, we propose KheOps, a collaborative environment specifically designed to enable cost-effective reproducibility and replicability of Edge-to-Cloud experiments. KheOps is composed of three core elements: (1) an experiment repository; (2) a notebook environment; and (3) a multi-platform experiment methodology. We illustrate KheOps with a real-life Edge-to-Cloud application. The evaluations explore the point of view of the authors of an experiment described in an article (who aim to make their experiments reproducible) and the perspective of their readers (who aim to replicate the experiment). The results show how KheOps helps authors to systematically perform repeatable and reproducible experiments on the Grid5000 + FIT IoT LAB testbeds. Furthermore, KheOps helps readers to cost-effectively replicate authors experiments in different infrastructures such as Chameleon Cloud + CHI@Edge testbeds, and obtain the same conclusions with high accuracies (> 88% for all performance metrics).