On Parameter Tying by Quantization
On Parameter Tying by Quantization
复制标题
DOI：
10.1609/aaai.v30i1.10429
复制
发表时间：
2016-02
期刊：
影响因子：
--
通讯作者：
Li Chou;Somdeb Sarkhel;Nicholas Ruozzi;Vibhav Gogate
中科院分区：
文献类型：
--
作者：
Li Chou;Somdeb Sarkhel;Nicholas Ruozzi;Vibhav Gogate
最大似然估计（MLE）通常是渐近一致的，但容易出现过度拟合。为了解决这个问题，实践中经常采用正则化方法，以（稍微）增加偏差为代价来减少方差。在本文中，我们提出了一种替代方差减少（正则化）技术，该技术将 MLE 估计量化为后处理步骤，从而产生具有多个绑定参数的更平滑的模型。我们为我们的新技术提供并证明了错误界限，并通过实验证明它通常会产生比使用 MLE 学习的模型具有更高测试集对数似然的模型。我们还提出了一种新的重要性采样算法，用于在具有多个绑定参数的模型中进行快速近似推理。我们的实验表明，在使用我们基于量化的方法学习的具有绑定参数的模型上，我们的新推理算法优于吉布斯采样和 MC-SAT 等现有方法。
The maximum likelihood estimator (MLE) is generally asymptotically consistent but is susceptible to over-fitting. To combat this problem, regularization methods which reduce the variance at the cost of (slightly) increasing the bias are often employed in practice. In this paper, we present an alternative variance reduction (regularization) technique that quantizes the MLE estimates as a post processing step, yielding a smoother model having several tied parameters. We provide and prove error bounds for our new technique and demonstrate experimentally that it often yields models having higher test-set log-likelihood than the ones learned using the MLE. We also propose a new importance sampling algorithm for fast approximate inference in models having several tied parameters. Our experiments show that our new inference algorithm is superior to existing approaches such as Gibbs sampling and MC-SAT on models having tied parameters, learned using our quantization-based approach.