Approximation Lasso Methods for Language Modeling
Approximation Lasso Methods for Language Modeling
复制标题
语言建模的近似套索方法
DOI：
10.3115/1220175.1220204
复制
发表时间：
2006
期刊：
影响因子：
--
通讯作者：
Bin Yu
中科院分区：
文献类型：
--
作者：
Jianfeng Gao;Hisami Suzuki;Bin Yu
Lasso 是线性模型中参数估计的正则化方法。它根据模型复杂性的损失函数优化模型参数。本文探讨了如何使用 lasso 进行文本输入的统计语言建模。由于参数数量非常多，直接优化惩罚套索损失函数是不可能的。因此，我们研究了两种近似方法，即增强套索（BLasso）和前向阶段线性回归（FSLR）。当与指数损失函数一起使用时，这两种方法与已用作语言建模判别训练方法的提升算法非常相似。对日语文本输入任务的评估表明，BLasso 能够产生对 lasso 解决方案的最佳近似，并在字符错误率、过度提升和传统最大似然估计方面带来显着改进。
Lasso is a regularization method for parameter estimation in linear models. It optimizes the model parameters with respect to a loss function subject to model complexities. This paper explores the use of lasso for statistical language modeling for text input. Owing to the very large number of parameters, directly optimizing the penalized lasso loss function is impossible. Therefore, we investigate two approximation methods, the boosted lasso (BLasso) and the forward stagewise linear regression (FSLR). Both methods, when used with the exponential loss function, bear strong resemblance to the boosting algorithm which has been used as a discriminative training method for language modeling. Evaluations on the task of Japanese text input show that BLasso is able to produce the best approximation to the lasso solution, and leads to a significant improvement, in terms of character error rate, over boosting and the traditional maximum likelihood estimation.