A Comparison of Algorithms for Maximum Entropy Parameter Estimation
A Comparison of Algorithms for Maximum Entropy Parameter Estimation
复制标题
DOI：
10.3115/1118853.1118871
复制
发表时间：
2002-08
期刊：
影响因子：
--
通讯作者：
Robert Malouf
中科院分区：
文献类型：
--
作者：
Robert Malouf
条件最大熵（ME）模型提供了一种通用机器学习技术，已成功应用于计算机视觉和计量经济学等领域，并用于自然语言处理中的各种分类问题。然而，ME模型的灵活性并非没有代价。虽然ME模型的参数估计在概念上是简单的，但在实践中，典型的自然语言任务的ME模型非常大，并且可能包含数千个自由参数。在本文中，我们考虑了一些估计ME模型参数的算法，包括迭代尺度法，梯度上升法，共轭梯度法和变尺度法。令人惊讶的是，标准使用的迭代缩放算法相比其他算法表现得很差，并且对于所有测试问题，有限内存可变度量算法优于其他选择。
Conditional maximum entropy (ME) models provide a general purpose machine learning technique which has been successfully applied to fields as diverse as computer vision and econometrics, and which is used for a wide variety of classification problems in natural language processing. However, the flexibility of ME models is not without cost. While parameter estimation for ME models is conceptually straightforward, in practice ME models for typical natural language tasks are very large, and may well contain many thousands of free parameters. In this paper, we consider a number of algorithms for estimating the parameters of ME models, including iterative scaling, gradient ascent, conjugate gradient, and variable metric methods. Sur-prisingly, the standardly used iterative scaling algorithms perform quite poorly in comparison to the others, and for all of the test problems, a limited-memory variable metric algorithm outperformed the other choices.