Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds
Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds
复制标题
适用于混合云上数据科学 Jupyter Notebooks 的上下文感知执行迁移工具
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
M. Netto
中科院分区：
文献类型：
--
作者：
Renato L. F. Cunha;L. V. Real;Renan Souza;B. Silva;M. Netto
交互式计算笔记本，例如Jupyter笔记本，已经成为开发和改进数据驱动模型的流行工具。这类笔记本往往在用户自己的机器或云环境中执行，这两种方法各有优缺点。本文提出了一个作为Jupyter扩展开发的解决方案，它可以自动选择哪些单元以及在哪些场景中，这些单元应该迁移到更合适的执行平台。我们描述了如何减少笔记本的执行状态以减少迁移时间，并探索了与笔记本的用户交互模式知识，以确定应该迁移哪些单元块。使用来自地球科学（遥感）、图像识别和手写数字识别（机器学习）的笔记本，我们的实验表明，当考虑到用户与笔记本的交互性时，笔记本状态减少高达55x，迁移决策导致性能提高高达3.25 x。
Interactive computing notebooks, such as Jupyter notebooks, have become a popular tool for developing and improving data-driven models. Such notebooks tend to be executed either in the user’s own machine or in a cloud environment, having drawbacks and benefits in both approaches. This paper presents a solution developed as a Jupyter extension that automatically selects which cells, as well as in which scenarios, such cells should be migrated to a more suitable platform for execution. We describe how we reduce the execution state of the notebook to decrease migration time and we explore the knowledge of user interactivity patterns with the notebook to determine which blocks of cells should be migrated. Using notebooks from Earth science (remote sensing), image recognition, and hand written digit identification (machine learning), our experiments show notebook state reductions of up to 55× and migration decisions leading to performance gains of up to 3.25× when the user interactivity with the notebook is taken into consideration.