Unsupervised Learning of Morphology

该项目致力于开发一个新系统，该系统可以从未注释的文本中同时发现世界上各种语言的单词形态和词性模式。 给定一定数量的训练文本，这样的系统将产生一个转换器，它将新文本中的单词分割成词干和词缀，并确定每个单词作为一个整体的词性。 通过无监督学习，迭代自举过程将联合收割机结合几个不同的语言知识源，以逐渐建立范式形式的语言表示。从这些范式中，提取符号化的词性规则和形态音位重写规则，然后将其编译成概率有限状态转换器，该转换器可以用形态和词性标记新的文本。尽管机器学习技术广泛应用于自然语言处理，但开发形态分析器仍然需要大量的人力。 虽然英语的形态学非常简单，但计算机对世界上大多数语言的文本或语音的自动分析取决于适当的形态分析器的可用性。 这也是重要的自动信息提取的重要问题，在生物医学领域，它是必要的，以分析复杂的结构的技术术语，即使在英语。 这种分析器在自然语言处理中的大多数应用中是有用的，包括解析、信息检索、机器翻译、文本摘要、语音合成中的正确发音、语音识别中的语言模型、语言生成和命名实体识别。

This project is working to develop a new system that simultaneously discovers the patterns of word morphology and parts of speech for a wide range of the world's languages from unannotated text. Given a quantity of training text, such a system will yield a transducer, which segments the words in new texts into stems and affixes and determine the part of speech of each word as a whole. Through unsupervised learning, an iterative bootstrapping procedure will combine several different linguistic knowledge sources to gradually build up a representation of the language in the form of paradigms. From these paradigms, symbolic part of speech rules and morphophonological rewrite rules will be extracted, which will then be compiled into a probabilistic finite-state transducer, which can label new texts with morphology and part of speech.Despite the widespread application of machine learning techniques to natural language processing, developing morphological analyzers still involves much human effort. While the morphology of English is very simple, the automatic analysis by computer of texts or speech in the majority of the world's languages depend on the availability of appropriate morphological analyzers. It is also important for the important problem of automatic information extraction in the biomedical domain, where it is necessary to analyze the complex structure of technical terms, even in English. Such analyzers are useful in most applications in natural language processing, including parsing, information retrieval, machine translation, text summarization, correct pronunciation in speech synthesis, language models in speech recognition, language generation, and named entity recognition.