Similarity Based Label Smoothing For Dialogue Generation
Similarity Based Label Smoothing For Dialogue Generation
复制标题
DOI：
--
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Sougata Saha;Souvik Das;R. Srihari
中科院分区：
文献类型：
--
作者：
Sougata Saha;Souvik Das;R. Srihari
生成式神经会话系统通常通过最小化训练“硬”目标和预测logit之间的熵损失来训练。性能增益和改进的泛化通常通过采用正则化技术（如标签平滑）来实现，该技术将训练“硬”目标转换为软目标。然而，标签平滑在不正确的训练目标上强制数据独立的均匀分布，导致等概率的错误假设。在本文中，我们提出并实验结合数据依赖的字相似性为基础的加权方法，将标签平滑的不正确的目标概率的均匀分布，以更现实的分布基于语义。我们引入超参数来控制不正确的目标分布，并在两个标准开放域对话语料库上使用标准标签平滑的损失训练网络，报告显着的性能增益。
Generative neural conversational systems are typically trained by minimizing the entropy loss between the training “hard” targets and the predicted logits. Performance gains and improved generalization are often achieved by employing regularization techniques like label smoothing, which converts the training “hard” targets to soft targets. However, label smoothing enforces a data independent uniform distribution on the incorrect training targets, leading to a false assumption of equiprobability. In this paper, we propose and experiment with incorporating data-dependent word similarity-based weighing methods to transform the uniform distribution of the incorrect target probabilities in label smoothing to a more realistic distribution based on semantics. We introduce hyperparameters to control the incorrect target distribution and report significant performance gains over networks trained using standard label smoothing-based loss on two standard open-domain dialogue corpora.