Validation and verification of regression in small data sets
Validation and verification of regression in small data sets
复制标题
DOI：
10.1016/s0169-7439(98)00167-1
复制
发表时间：
1998-12-14
影响因子：
3.9
通讯作者：
Dardenne, P
中科院分区：
文献类型：
--
作者：
Martens, HA;Dardenne, P
比较了在多变量建模中使用小数据集的四种不同方法。从长远来看，预测的准确性。这种情况下的建模涉及多变量校准：(Y)OVER CAP=f(X)。这项研究包括在一个真实数据的大型数据库中进行蒙特卡罗模拟；X=近红外反射光谱，Y=蛋白质百分比，在922个玉米植株样本中测量。重复从数据库中随机选择小数据集(40-120个对象)，每次模拟仅有小样本集可用于估计、优化和评估校准模型的情况。“真实”的表观预测误差每次都在剩余的数据库中进行控制。为了研究四种不同验证方法的统计性能，重复了100次。在每次蒙特卡罗重复中，将可用数据集分割成校准集和测试集与完全交叉验证进行比较。结果表明，将样本从已经有限的可用样本集中移到独立的验证测试集中，严重降低了校准模型的预测性能，同时给出了对模型预测性能的不确定的、系统的过度乐观的评估。完全交叉验证提供了更好的预测性能，并且对这种预测性能只给出了略微过于乐观的评估。进一步删除更多可用于独立验证测试集的样本，从长期来看是正确的，尽管对校准模型的预测性能的估计不确定，但这一性能水平已经严重恶化。用交叉验证的方法对模型的预测性能进行交替验证，得到的结果与交叉验证的结果非常相似。这些来自真实数据的结果与之前对人工模拟数据的发现非常一致。结果表明，完全交叉验证优于使用独立验证测试集和独立验证测试集。(C)1998 Elsevier Science B.V.，保留所有权利。
Four different methods of using small data sets in multivariate modelling are compared w.r.t. predictive precision in the long-run. The modelling in this case concerns multivariate calibration: (y) over cap=f(X). The study consists of a Monte Carlo simulation within a large data base of real data; X = NIR reflectance spectra and y = protein percentage, measured in 922 whole maize plant samples. Small data sets (40-120 objects) were repeatedly selected at random from the data base, each time simulating the situation of having only a small set of samples available for estimating, optimizing and assessing the calibration model. The 'true' apparent prediction error was each time controlled in the remaining data base. This was replicated 100 times in order to study the statistical performance of the four different validation methods. In each Monte Carlo replicate, the splitting of the available data set into calibration set and test set was compared to full cross validation. The results demonstrated that removing samples from an already Limited set of available samples to an independent VALIDATION TEST SET seriously reduced the predictive performance of the calibrated models, and at the same time gave uncertain, systematically over-optimistic assessment of the models' predictive performance. Full CROSS VALIDATION gave improved predictive performance, and gave only slightly over-optimistic assessment of this predictive performance. Further removal of even more of the available samples for use in an independent VERIFICATION TEST SET gave in-the-long-run correct, although uncertain estimates of the predictive performance of the calibrated models, but this performance level had seriously deteriorated. Alternative verification of the model's predictive performance by the method of CROSS VERIFICATION gave results very similar to those of the cross validation. These results from real data correspond closely to previous findings for artificially simulated data. It appears that full cross validation is superior to both the use of independent validation test set and independent verification test set. (C) 1998 Elsevier Science B.V, All rights reserved.