Versioning for End-to-End Machine Learning Pipelines
Versioning for End-to-End Machine Learning Pipelines
复制标题
端到端机器学习管道的版本控制
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
T. V. Kasteren
中科院分区：
文献类型：
--
作者：
T. V. D. Weide;D. Papadopoulos;O. Smirnov;Michal Zielinski;T. V. Kasteren
在共享环境中运行的端到端机器学习管道很难实现。生产管道通常由多个相互依赖的处理阶段组成。在阶段之间，中间结果被持久化以减少冗余计算并提高鲁棒性。这些结果可能以数据集的形式出现在数据处理管道中，或者以模型系数的形式出现在模型训练管道中。重用持久化结果可以提高效率，但同时也会产生复杂的依赖关系。每当其中一个处理阶段发生变化时，无论是由于代码更改还是由于参数更改，都很难发现哪些数据集可以重用，哪些数据集应该重新计算。在本文中，我们以以前的工作为基础，生成数据集的派生，以确保管道的多个版本可以并行运行，同时最大限度地减少冗余计算量。我们的扩展包括简化导航和重用的部分派生，对管道模式更改的显式支持，以及运行管道的中央注册中心，以协调团队之间的管道升级。
End-to-end machine learning pipelines that run in shared environments are challenging to implement. Production pipelines typically consist of multiple interdependent processing stages. Between stages, the intermediate results are persisted to reduce redundant computation and to improve robustness. Those results might come in the form of datasets for data processing pipelines or in the form of model coefficients in case of model training pipelines. Reusing persisted results improves efficiency but at the same time creates complicated dependencies. Every time one of the processing stages is changed, either due to code change or due to parameters change, it becomes difficult to find which datasets can be reused and which should be recomputed. In this paper we build upon previous work to produce derivations of datasets to ensure that multiple versions of a pipeline can run in parallel while minimizing the amount of redundant computations. Our extensions include partial derivations to simplify navigation and reuse, explicit support for schema changes of pipelines, and a central registry of running pipelines to coordinate upgrading pipelines between teams.