TERA: optimizing stochastic regression tests in machine learning projects
TERA: optimizing stochastic regression tests in machine learning projects
复制标题
DOI：
10.1145/3460319.3464844
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Saikat Dutta;Jeeva Selvam;Aryaman Jain;Sasa Misailovic
中科院分区：
文献类型：
--
作者：
Saikat Dutta;Jeeva Selvam;Aryaman Jain;Sasa Misailovic
许多机器学习（ML）算法的随机性使得ML工具和库的测试具有挑战性。ML算法允许开发人员通过一组超参数来控制其准确性和运行时间，这些超参数通常在测试中手动选择。这种选择通常过于保守，导致测试执行缓慢，从而增加了回归测试的成本。我们提出了TERA，这是第一个自动化技术，用于降低机器学习工具和库（统称为项目）中回归测试的成本，而不会使测试更加复杂。TERA解决了探索测试执行时间和其脆弱性之间的权衡空间的问题，作为算法超参数空间上的随机优化的一个实例。TERA介绍了如何利用统计收敛测试技术来估计在优化过程中针对特定超参数选择的测试的脆弱程度。我们使用从15个流行机器学习项目中选出的160个测试的语料库来评估TERA。总的来说，TERA在原始测试中获得了2.23倍的地理平均加速，最小通过概率阈值为99%。我们还表明，新的测试并没有减少故障检测能力，通过突变研究和研究一组12个历史的构建失败的研究项目。
The stochastic nature of many Machine Learning (ML) algorithms makes testing of ML tools and libraries challenging. ML algorithms allow a developer to control their accuracy and run-time through a set of hyper-parameters, which are typically manually selected in tests. This choice is often too conservative and leads to slow test executions, thereby increasing the cost of regression testing. We propose TERA, the first automated technique for reducing the cost of regression testing in Machine Learning tools and libraries(jointly referred to as projects) without making the tests more flaky. TERA solves the problem of exploring the trade-off space between execution time of the test and its flakiness as an instance of Stochastic Optimization over the space of algorithm hyper-parameters. TERA presents how to leverage statistical convergence-testing techniques to estimate the level of flakiness of the test for a specific choice of hyper-parameters during optimization. We evaluate TERA on a corpus of 160 tests selected from 15 popular machine learning projects. Overall, TERA obtains a geo-mean speedup of 2.23x over the original tests, for the minimum passing probability threshold of 99%. We also show that the new tests did not reduce fault detection ability through a mutation study and a study on a set of 12 historical build failures in studied projects.