A survey on addressing high-class imbalance in big data
A survey on addressing high-class imbalance in big data
复制标题
DOI：
10.1186/s40537-018-0151-6
复制
发表时间：
2018-11-01
影响因子：
8.1
通讯作者：
Seliya, Naeem
中科院分区：
文献类型：
--
作者：
Leevy, Joffrey L.;Khoshgoftaar, Taghi M.;Seliya, Naeem
在多数-少数分类问题中，数据集中的类不平衡会极大地扭曲分类器的性能，引入对多数类的预测偏差。假设阳性（少数）类是感兴趣的组，并且给定的应用程序领域规定假阴性比假阳性代价高得多，则阴性（多数）类预测偏差可能会产生不利后果。对于大数据，由于相对更大的数据集结构多样且复杂，因此减轻类别不平衡带来了更大的挑战。本文对近8年来发表的研究进行了大量调查，重点关注大数据中的高级班级失衡（即多数与少数班级比例介于100:1和10,000:1之间），以评估在解决班级失衡带来的不利影响方面的最新进展。本文涵盖了两种技术，包括数据级（例如，数据采样）和算法级（例如，成本敏感和混合/集成）方法。数据抽样方法在解决类失衡方面很受欢迎，随机过采样方法通常表现出更好的总体结果。在算法层面，有一些杰出的表演者。然而，在已发表的研究中，存在不一致和相互矛盾的结果，加上评估技术的范围有限，表明需要进行更全面的比较研究。
In a majority-minority classification problem, class imbalance in the dataset(s) can dramatically skew the performance of classifiers, introducing a prediction bias for the majority class. Assuming the positive (minority) class is the group of interest and the given application domain dictates that a false negative is much costlier than a false positive, a negative (majority) class prediction bias could have adverse consequences. With big data, the mitigation of class imbalance poses an even greater challenge because of the varied and complex structure of the relatively much larger datasets. This paper provides a large survey of published studies within the last 8 years, focusing on high-class imbalance (i.e., a majority-to-minority class ratio between 100:1 and 10,000:1) in big data in order to assess the state-of-the-art in addressing adverse effects due to class imbalance. In this paper, two techniques are covered which include Data-Level (e.g., data sampling) and Algorithm-Level (e.g., cost-sensitive and hybrid/ensemble) Methods. Data sampling methods are popular in addressing class imbalance, with Random Over-Sampling methods generally showing better overall results. At the Algorithm-Level, there are some outstanding performers. Yet, in the published studies, there are inconsistent and conflicting results, coupled with a limited scope in evaluated techniques, indicating the need for more comprehensive, comparative studies.