Being Properly Improper
Being Properly Improper
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
R. Nock;Tyler Sypherd;L. Sankar
中科院分区：
文献类型：
--
作者：
R. Nock;Tyler Sypherd;L. Sankar
监督损失的适用性规定损失函数将学习算法塑造为数据生成分布的真实后验。不幸的是，现代机器学习中的数据可能会以多种方式被破坏或扭曲。因此，在扭曲数据上优化适当的损失函数可能会危险地导致学习算法走向扭曲的后验，而不是期望的干净后验。许多论文应对特定的扭曲（例如，标签/特征/对抗性噪音），但越来越需要在正确性之上建立统一且可操作的理解。我们的主要理论贡献是对正确性框架的概括，其概念称为扭曲正确性，它描述了具有将扭曲后验“解开”为干净后验的能力的损失函数。值得注意的是，我们证明了损失函数的一个非平凡的扩展，称为$\alpha$-loss，它首先在信息论中引入，是扭曲正确的。我们研究了一种名为 PILBoost 的新型增强算法下的扭曲固有 $\alpha$-loss，并为该算法提供了正式的实验结果。我们总体的实际结论是，在扭曲数据的几个变体上，扭曲适当的 $\alpha$-loss 优于适当的 $\log$-loss。
Properness for supervised losses stipulates that the loss function shapes the learning algorithm towards the true posterior of the data generating distribution. Unfortunately, data in modern machine learning can be corrupted or twisted in many ways. Hence, optimizing a proper loss function on twisted data could perilously lead the learning algorithm towards the twisted posterior, rather than to the desired clean posterior. Many papers cope with specific twists (e.g., label/feature/adversarial noise), but there is a growing need for a unified and actionable understanding atop properness. Our chief theoretical contribution is a generalization of the properness framework with a notion called twist-properness, which delineates loss functions with the ability to"untwist"the twisted posterior into the clean posterior. Notably, we show that a nontrivial extension of a loss function called $\alpha$-loss, which was first introduced in information theory, is twist-proper. We study the twist-proper $\alpha$-loss under a novel boosting algorithm, called PILBoost, and provide formal and experimental results for this algorithm. Our overarching practical conclusion is that the twist-proper $\alpha$-loss outperforms the proper $\log$-loss on several variants of twisted data.