MLDemon: Deployment Monitoring for Machine Learning Systems
MLDemon: Deployment Monitoring for Machine Learning Systems
复制标题
DOI：
--
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Antonio A. Ginart;Martin Jinye Zhang;James Y. Zou
中科院分区：
文献类型：
--
作者：
Antonio A. Ginart;Martin Jinye Zhang;James Y. Zou
ML系统的部署后监测对于确保可靠性至关重要，特别是在新用户输入可能不同于培训分发的情况下。在这里，我们提出了一种新的方法MLDemon，用于ML部署监控。MLDemon集成了未标记的数据和少量的按需标签，以实时估计ML模型在给定数据流上的当前性能。在预算限制的情况下，MLDemon决定何时获得额外的、可能昂贵的专家监督标签来验证模型。在具有不同分布漂移和模型的时态数据集上，MLDemon的性能优于现有方法。此外，我们还提供了理论分析，表明MLDemon对于一大类分布漂移是最优的极小极大速率。
Post-deployment monitoring of ML systems is critical for ensuring reliability, especially as new user inputs can differ from the training distribution. Here we propose a novel approach, MLDemon, for ML DEployment MONitoring. MLDemon integrates both unlabeled data and a small amount of on-demand labels to produce a real-time estimate of the ML model's current performance on a given data stream. Subject to budget constraints, MLDemon decides when to acquire additional, potentially costly, expert supervised labels to verify the model. On temporal datasets with diverse distribution drifts and models, MLDemon outperforms existing approaches. Moreover, we provide theoretical analysis to show that MLDemon is minimax rate optimal for a broad class of distribution drifts.