Facing Imbalanced Data Recommendations for the Use of Performance Metrics.
Facing Imbalanced Data Recommendations for the Use of Performance Metrics.
复制标题
DOI：
10.1109/acii.2013.47
复制
发表时间：
2013
期刊：
影响因子：
--
通讯作者：
De La Torre F
中科院分区：
文献类型：
--
作者：
Jeni LA;Cohn JF;De La Torre F
识别面部动作单元（AU）对于情境分析和自动视频注释非常重要。以前的工作强调人脸跟踪和注册和特征分类器的选择。相对被忽视的是不平衡的数据对动作单元检测的影响。虽然机器学习社区已经意识到训练分类器的偏斜数据问题，但很少关注偏斜如何影响性能指标。为了解决这个问题，我们进行了实验，使用模拟分类器和三个主要的数据库，不同的大小，类型的FACS编码，和程度的倾斜。我们评估了偏斜对阈值指标（准确度、F评分、Cohen's kappa和Krippendorf's alpha）和秩指标（受试者工作特征（ROC）曲线下面积和精确度-召回曲线）的影响。除ROC曲线下面积外，所有指标均因偏态分布而衰减，在许多情况下，偏态分布的衰减幅度非常大。虽然ROC不受偏斜的影响，但精确度-召回率曲线表明ROC可能掩盖了糟糕的表现。我们的研究结果表明，偏斜是评估性能指标的一个关键因素。为了避免或最大限度地减少偏置性能估计，我们建议报告偏置归一化分数沿着获得的分数。
Recognizing facial action units (AUs) is important for situation analysis and automated video annotation. Previous work has emphasized face tracking and registration and the choice of features classifiers. Relatively neglected is the effect of imbalanced data for action unit detection. While the machine learning community has become aware of the problem of skewed data for training classifiers, little attention has been paid to how skew may bias performance metrics. To address this question, we conducted experiments using both simulated classifiers and three major databases that differ in size, type of FACS coding, and degree of skew. We evaluated influence of skew on both threshold metrics (Accuracy, F-score, Cohen's kappa, and Krippendorf's alpha) and rank metrics (area under the receiver operating characteristic (ROC) curve and precision-recall curve). With exception of area under the ROC curve, all were attenuated by skewed distributions, in many cases, dramatically so. While ROC was unaffected by skew, precision-recall curves suggest that ROC may mask poor performance. Our findings suggest that skew is a critical factor in evaluating performance metrics. To avoid or minimize skew-biased estimates of performance, we recommend reporting skew-normalized scores along with the obtained ones.