Machine learning algorithm validation with a limited sample size
Machine learning algorithm validation with a limited sample size
复制标题
DOI：
10.1371/journal.pone.0224365
复制
发表时间：
2019-11-07
期刊：
影响因子：
3.7
通讯作者：
Casson, Alexander J.
中科院分区：
文献类型：
--
作者：
Vabalas, Andrius;Gowen, Emma;Casson, Alexander J.
神经成像、基因组、运动跟踪、眼球跟踪和许多其他基于技术的数据收集方法的进步催生了大量高维数据集，由于涉及人类参与者的数据收集固有的高成本，这些数据集通常只有少量样本。少量样本的高维数据对于识别生物标志物以及进行可行性和试点工作至关重要，但它可能会导致机器学习 (ML) 性能估计出现偏差。我们对应用机器学习从非自闭症个体中预测自闭症的研究进行的回顾表明，较小的样本量与较高的报告分类准确性相关。因此，我们研究了这种偏差是否可能是由于使用不能充分控制过度拟合的验证方法引起的。我们的模拟表明，K 折交叉验证 (CV) 在样本量较小的情况下会产生强烈偏差的性能估计，并且在样本量为 1000 时偏差仍然很明显。无论样本大小如何，嵌套 CV 和训练/测试分割方法都会产生稳健且无偏差的性能估计。我们还表明，如果对汇集的训练和测试数据执行特征选择，则比参数调整对偏差的影响要大得多。此外，还探讨了数据维度、超参数空间和 CV 折叠数量对偏差的影响，并将验证方法与可辨别的数据进行了比较。结果表明，在处理小型数据集时如何设计稳健的测试方法，以及如何根据使用的验证方法解释其他研究的结果。
Advances in neuroimaging, genomic, motion tracking, eye-tracking and many other technology-based data collection methods have led to a torrent of high dimensional datasets, which commonly have a small number of samples because of the intrinsic high cost of data collection involving human participants. High dimensional data with a small number of samples is of critical importance for identifying biomarkers and conducting feasibility and pilot work, however it can lead to biased machine learning (ML) performance estimates. Our review of studies which have applied ML to predict autistic from non-autistic individuals showed that small sample size is associated with higher reported classification accuracy. Thus, we have investigated whether this bias could be caused by the use of validation methods which do not sufficiently control overfitting. Our simulations show that K-fold Cross-Validation (CV) produces strongly biased performance estimates with small sample sizes, and the bias is still evident with sample size of 1000. Nested CV and train/test split approaches produce robust and unbiased performance estimates regardless of sample size. We also show that feature selection if performed on pooled training and testing data is contributing to bias considerably more than parameter tuning. In addition, the contribution to bias by data dimensionality, hyper-parameter space and number of CV folds was explored, and validation methods were compared with discriminable data. The results suggest how to design robust testing methodologies when working with small datasets and how to interpret the results of other studies based on what validation method was used.