Performance Evaluation Indicator (PEI): A new paradigm to evaluate the competence of machine learning classifiers in predicting rockmass conditions
Performance Evaluation Indicator (PEI): A new paradigm to evaluate the competence of machine learning classifiers in predicting rockmass conditions
复制标题
DOI：
10.1016/j.aei.2020.101232
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Mengqi Zhu;M. Gutierrez;Hehua Zhu;J. Ju;S. Sarna
中科院分区：
文献类型：
--
作者：
Mengqi Zhu;M. Gutierrez;Hehua Zhu;J. Ju;S. Sarna
为了说明建立在专有数据库上的机器学习分类器之间的无偏见比较，并保证这些分类器的有效性和健壮性，本研究提出了一个性能评价指标(PEI)和相应的失效准则。在一个排水TBM项目的四个数据集上训练了三种类型的机器学习分类器，包括严格二进制分类器、正常多类分类器和误分类代价敏感分类器。结果表明：(1)PEI通过隔离不同类别间重叠度的影响，成功地比较了不同场景下不同分类器的分类能力；(2)当类别间重叠度大于8：1时，代价敏感算法适用于岩体分类。本研究的贡献在于填补了分类器在训练数据不平衡情况下性能评估的空白，并确定了该分类器的最佳应用场合。
To illustrate an unprejudiced comparison among machine learning classifiers established on proprietary databases, and to guarantee the validity and robustness of these classifiers, a Performance Evaluation Indicator (PEI) and the corresponding failure criterion are proposed in this study. Three types of machine learning classifiers, including the strictly binary classifier, the normal multiclass classifier and the misclassification cost-sensitive classifier, are trained on four datasets recorded from a water drainage TBM project. The results indicate that: (1) the PEI successfully compares the competence of classifiers under different scenarios by isolating the effects of different overlapping-degree of rockmass classes, and (2) the cost-sensitive algorithm is warranted to classify rockmasses when the ratio of inter-class classes is more than 8:1. The contributions of this research are to fill the gap in performance evaluations of a classifier for imbalanced training data, and to identify the best situation to apply this classifier.