Improved Statistical Language Models

这是一项为期三年的持续拨款的第一年奖励，该拨款用于研究概率和概率语言模型在无上下文解析应用程序中的使用。本研究包括四个部分：1)将概率应用于英语句法分析，通过对手工标注的语料库应用标准统计技术自动生成概率语法；2)统计分析中的一部分资源涉及词的共现，这提高了语言模型的有效性，从而有助于在解析中确定，例如，一个名词比另一个名词更有可能出现在某个位置，而不仅仅是通过经典解析器获得的词性结果；3)由于统计参数被平滑，预计通过将具有相似统计量的单词分组在类中可能实现某种程度的语义泛化；最后，通过单字交叉熵等客观质量指标对得到的平滑模型进行严格检验。目标是改进目前在语音识别中使用的现有三元组模型，并朝着更好的语言理解系统努力。

9319516 Charniak This is the first-year award of a three-year continuing grant to study the use of probabilities and probabilistic language models in context-free parsing applications. The research consists of four parts: 1) use of probabilities in the syntactic parsing of English through automatic generation of a probabilistic grammar by applying standard statistical techniques to corpora that has been hand- tagged; 2) a portion of the resources in the statistical analysis involves the co-occurrence of words, which improves the effectiveness of the language model thereby helping the determination in the parse that, for example, a noun would be more likely in a certain position than another, rather than just the parts-of-speech results that would be obtained via a classical parser; 3) it is anticipated that some level of semantic generalization may be possible by grouping the words with similar statistics in classes, as statistical parameters are smoothed; and finally, 4) the model obtained and smoothed is rigorously tested through an objective quality measure such as the per-word cross entropy. The goal is to improve upon existing trigram models, currently in use in speech recognition, and to work towards better language understanding systems.