Real External Predictivity of QSAR Models: How To Evaluate It? Comparison of Different Validation Criteria and Proposal of Using the Concordance Correlation Coefficient
Real External Predictivity of QSAR Models: How To Evaluate It? Comparison of Different Validation Criteria and Proposal of Using the Concordance Correlation Coefficient
复制标题
DOI：
10.1021/ci200211n
复制
发表时间：
2011-09-01
影响因子：
5.6
通讯作者：
Gramatica, Paola
中科院分区：
文献类型：
--
作者：
Chirico, Nicola;Gramatica, Paola
QSAR模型的主要用途是预测新化学品的活性/性质的能力，这种外部预测能力通过各种验证标准进行评估。作为这种评价的一种衡量标准，经合组织准则提出了预测平方相关系数Q（F1）（2）（Shi等人）。然而，其他作者提出了其他验证标准：Golbraikh-Tropsha方法，r（m）（2），（Roy），Q（F2）（2）（Schuurmann等人），Q（F3）（2）（Consonni et al.）.在定量构效关系研究中，这些措施通常是一致的，尽管情况并非总是如此，因此，当获得相互矛盾的结果时，可能会引起怀疑。很可能上述标准在每种情况下都不是最好的，因此这里提出了使用模拟数据集的比较研究，使用支持者建议的阈值或QSAR建模中广泛使用的阈值。此外，一个不同的和简单的外部验证措施，一致性相关系数（CCC），提出并与其他标准进行了比较。大量的数据集被用来研究验证措施的一般行为，一致性相关系数被证明是最严格的。在使用更现实规模的模拟数据集时，发现CCC与其他验证措施在接受模型作为预测方面大致一致，约96%的时间，并且在几乎所有的例子中，它都是最谨慎的。建议的一致性相关系数也适用于真实的数据集，它似乎更稳定，并有助于在验证措施发生冲突时做出决策。由于它在概念上是简单的，并考虑到其稳定性和限制性，我们提出了一致性相关系数作为一种补充，或替代，更谨慎的措施，QSAR模型是外部预测。
The main utility of QSAR models is their ability to predict activities/properties for new chemicals, and this external prediction ability is evaluated by means of various validation criteria. As a measure for such evaluation the OECD guidelines have proposed the predictive squared correlation coefficient Q(F1)(2) (Shi et al.). However, other validation criteria have been proposed by other authors: the Golbraikh-Tropsha method, r(m)(2), (Roy), Q(F2)(2) (Schuurmann et al.), Q(F3)(2) (Consonni et al.). In QSAR studies these measures are usually in accordance, though this is not always the case, thus doubts can arise when contradictory results are obtained. It is likely that none of the aforementioned criteria is the best in every situation, so a comparative study using simulated data sets is proposed here, using threshold values suggested by the proponents or those widely used in QSAR modeling. In addition, a different and simple external validation measure, the concordance correlation coefficient (CCC), is proposed and compared with other criteria. Huge data sets were used to study the general behavior of validation measures, and the concordance correlation coefficient was shown to be the most restrictive. On using simulated data sets of a more realistic size, it was found that CCC was broadly in agreement, about 96% of the time, with other validation measures in accepting models as predictive, and in almost all the examples it was the most precautionary. The proposed concordance correlation coefficient also works well on real data sets, where it seems to be more stable, and helps in making decisions when the validation measures are in conflict. Since it is conceptually simple, and given its stability and restrictiveness, we propose the concordance correlation coefficient as a complementary, or alternative, more prudent measure of a QSAR model to be externally predictive.