Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity
Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity
复制标题
DOI：
--
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Dixian Zhu;Yiming Ying;Tianbao Yang
中科院分区：
文献类型：
--
作者：
Dixian Zhu;Yiming Ying;Tianbao Yang
我们研究了一个家庭的损失函数命名为标签分布鲁棒（LDR）损失的多类分类，制定从分布鲁棒优化（DRO）的角度来看，在给定的标签信息的不确定性建模和捕获的最坏情况下的分布权重。这种观点的好处有几个方面：（i）它提供了一个统一的框架来解释经典的交叉熵（CE）损失和SVM损失及其变体，（ii）它包括一个特殊的家庭对应的温度标度CE损失，这是广泛采用的，但了解甚少;（iii）它允许我们实现自适应的标签信息的不确定程度在一个实例级别。我们的贡献包括：（1）通过建立top-$k$模型，研究了一致性和鲁棒性（$\forall k\geq 1$）的多类分类的LDR损失的一致性，以及一个否定的结果，即一个top-$1$一致的和对称的鲁棒损失不能同时达到所有$k\geq 2$的top-$k$一致;（2）提出了一种新的自适应LDR损失算法，该算法根据每个实例的类标签噪声程度自动调整个体化的温度参数;（3）在6个噪声标签和1个干净设置下的7个基准数据集上对13个损失函数和一个真实世界的噪声数据集，我们证明了所提出的自适应LDR损失的稳定和有竞争力的性能。该代码在\url{https：//github.com/Optimization-AI/ICML2023_LDR}上开源。
We study a family of loss functions named label-distributionally robust (LDR) losses for multi-class classification that are formulated from distributionally robust optimization (DRO) perspective, where the uncertainty in the given label information are modeled and captured by taking the worse case of distributional weights. The benefits of this perspective are several fold: (i) it provides a unified framework to explain the classical cross-entropy (CE) loss and SVM loss and their variants, (ii) it includes a special family corresponding to the temperature-scaled CE loss, which is widely adopted but poorly understood; (iii) it allows us to achieve adaptivity to the uncertainty degree of label information at an instance level. Our contributions include: (1) we study both consistency and robustness by establishing top-$k$ ($\forall k\geq 1$) consistency of LDR losses for multi-class classification, and a negative result that a top-$1$ consistent and symmetric robust loss cannot achieve top-$k$ consistency simultaneously for all $k\geq 2$; (2) we propose a new adaptive LDR loss that automatically adapts the individualized temperature parameter to the noise degree of class label of each instance; (3) we demonstrate stable and competitive performance for the proposed adaptive LDR loss on 7 benchmark datasets under 6 noisy label and 1 clean settings against 13 loss functions, and on one real-world noisy dataset. The code is open-sourced at \url{https://github.com/Optimization-AI/ICML2023_LDR}.