Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)
Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)
复制标题
DOI：
10.1111/j.1365-2664.2006.01214.x
复制
发表时间：
2006-12-01
影响因子：
5.7
通讯作者：
Kadmon, Ronen
中科院分区：
文献类型：
--
作者：
Allouche, Omri;Tsoar, Asaf;Kadmon, Ronen
1. 近年来，生态学家和保护管理者对物种分布模型的使用大幅增加，同时也意识到需要为此类模型的预测提供准确性评估。 kappa 统计量是最广泛使用的衡量生成存在-不存在预测的模型性能的指标，但一些研究批评它本质上依赖于患病率，并认为这种依赖关系在预测准确性的估计中引入了统计假象。这种批评最近得到了计算机模拟的支持，显示 kappa 以单峰方式对模型物种的流行做出反应。2。在本文中，我们为观察到的 kappa 对流行率的依赖性提供了理论解释，并向生态学引入了另一种准确性测量方法，即真实技能统计（TSS），它纠正了这种依赖性，同时仍然保留了 kappa 的所有优点。我们还通过对以色列 128 种木本植物的分布模式进行建模，使用经验数据比较了 kappa 和 TSS 对流行率的响应。3。理论分析表明，kappa 以单峰方式响应患病率的变化，并且使 kappa 最大化的患病率水平取决于敏感性（正确预测存在的比例）和特异性（正确预测缺席的比例）之间的比率。相反，TSS 与患病率无关。4．当使用经验数据比较这两种准确性指标时，kappa 显示出对患病率的单峰响应，与理论分析一致。 TSS 显示出对流行率的线性响应递减，我们将这一结果解释为反映了真实的生态现象，而不是统计假象。这种解释得到了以下事实的支持：ROC 曲线下面积也发现了类似的模式，该曲线已知与患病率无关5。合成与应用。我们的结果提供了理论和经验证据，表明 kappa 是生态学中使用最广泛的模型性能衡量标准之一，但存在严重的局限性，使其不适合此类应用。我们建议的替代方案 TSS 弥补了 kappa 的缺点，同时保留了其所有优点。因此，当预测表示为存在-不存在图时，我们建议将 TSS 作为一种简单直观的衡量物种分布模型性能的方法。
1. In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence-absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion.2. In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel.3. The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence.4. When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence.5. Synthesis and applications. Our results provide theoretical and empirical evidence that kappa, one of the most widely used measures of model performance in ecology, has serious limitations that make it unsuitable for such applications. The alternative we suggest, TSS, compensates for the shortcomings of kappa while keeping all of its advantages. We therefore recommend the TSS as a simple and intuitive measure for the performance of species distribution models when predictions are expressed as presence-absence maps.