The STONE Curve: A ROC-Derived Model Performance Assessment Tool.
The STONE Curve: A ROC-Derived Model Performance Assessment Tool.
复制标题
DOI：
10.1029/2020ea001106
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
Rastätter L
中科院分区：
文献类型：
--
作者：
Liemohn MW;Azari AR;Ganushkina NY;Rastätter L
介绍了一种新的模型验证和性能评估工具--数值评估曲线观测值滑动阈值（STONE）。它基于相对工作特征（ROC）曲线技术，但STONE工具使用观测值的连续性，而不是将所有观测值按类别分类。不是在观察中定义事件，然后仅在分类器/模型数据集中滑动阈值，而是同时为观察值和模型值改变阈值，其中数据和模型具有相同的阈值。只有当观测值是连续的，并且模型输出与观测值采用相同的单位和尺度时，这才是可能的，也就是说，模型试图精确地重现数据。STONE曲线与ROC曲线绘制的检测概率与错误检测概率有几个相似之处，范围从低阈值的（1，1）角到高阈值的（0，0）角，零截距单位斜率线以上的值表明优于随机预测能力。主要区别在于STONE曲线可以是非单调的，在x和y方向上都加倍。这些涟漪揭示了数据-模型值对中的不对称性。这种新技术被应用到模拟输出的一个共同的地磁活动指数，以及高能电子通量在地球的内磁层。它不仅限于空间物理应用，还可用于任何科学或工程领域，其中使用数值模型来重现观测结果。提出了一种新的基于事件检测的模型性能评价指标，该指标在观测值和模型值中均采用滑动阈值。新指标类似于相对工作特征曲线，但使用的是连续观测值，而不仅仅是分类状态。新指标用于常见地磁活动参数的真实的时间模型预测，展示了其特点和优势。
A new model validation and performance assessment tool is introduced, the sliding threshold of observation for numeric evaluation (STONE) curve. It is based on the relative operating characteristic (ROC) curve technique, but instead of sorting all observations in a categorical classification, the STONE tool uses the continuous nature of the observations. Rather than defining events in the observations and then sliding the threshold only in the classifier/model data set, the threshold is changed simultaneously for both the observational and model values, with the same threshold value for both data and model. This is only possible if the observations are continuous and the model output is in the same units and scale as the observations, that is, the model is trying to exactly reproduce the data. The STONE curve has several similarities with the ROC curve—plotting probability of detection against probability of false detection, ranging from the (1,1) corner for low thresholds to the (0,0) corner for high thresholds, and values above the zero‐intercept unity‐slope line indicating better than random predictive ability. The main difference is that the STONE curve can be nonmonotonic, doubling back in both the x and y directions. These ripples reveal asymmetries in the data‐model value pairs. This new technique is applied to modeling output of a common geomagnetic activity index as well as energetic electron fluxes in the Earth's inner magnetosphere. It is not limited to space physics applications but can be used for any scientific or engineering field where numerical models are used to reproduce observations. A new event‐detection‐based metric for model performance appraisal is given with sliding thresholds in both observational and model values The new metric is like the relative operating characteristic curve but uses continuous observational values, not just categorical status The new metric is used on real‐time model predictions of common geomagnetic activity parameters, demonstrating its features and strengths