Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss
Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss
复制标题
DOI：
10.1109/tit.2019.2958705
复制
发表时间：
2018-10
影响因子：
2.5
通讯作者：
Amichai Painsky;G. Wornell
中科院分区：
文献类型：
--
作者：
Amichai Painsky;G. Wornell
损失函数度量给定数据实例的真实值与其估计拟合之间的差异。在分类问题中，如果期望损失的最小值是真实的潜在概率，则损失函数被称为是适当的。我们发现，对于二进制分类，与光滑，适当的，凸损失函数的分歧是上界的Kullback-Leibler（KL）分歧，在一个归一化常数。这意味着，通过最小化与KL散度相关的对数损失，我们可以最小化该集合中任何损失选择的上限。因此，对数损失是普遍的意义上提供性能保证方面的广泛类别的准确性措施。重要的是，这种普遍性的概念不是特定于问题的，使其能够用于各种应用，包括预测建模，数据聚类和样本复杂性分析。任意有限字母表的推广也得到了发展。所得不等式推广了几个著名的f-发散结果。
A loss function measures the discrepancy between the true values and their estimated fits, for a given instance of data. In classification problems, a loss function is said to be proper if a minimizer of the expected loss is the true underlying probability. We show that for binary classification, the divergence associated with smooth, proper, and convex loss functions is upper bounded by the Kullback-Leibler (KL) divergence, to within a normalization constant. This implies that by minimizing the logarithmic loss associated with the KL divergence, we minimize an upper bound to any choice of loss from this set. As such the logarithmic loss is universal in the sense of providing performance guarantees with respect to a broad class of accuracy measures. Importantly, this notion of universality is not problem-specific, enabling its use in diverse applications, including predictive modeling, data clustering and sample complexity analysis. Generalizations to arbitary finite alphabets are also developed. The derived inequalities extend several well-known $f$ -divergence results.