Word form similarity computation and application in natural language processing

点击翻译按钮获取中文摘要

In natural language processing, words are often treated as abstract entities divorced from their actual form. However, the actual orthographic or phonetic form of words is an important feature that can be utilized in various tasks. The algorithms that I previously developed for computing word similarity have proven to be effective in a variety of applications, including identification of confusable drug names, and matching person names across different scripts. In the current proposal, I describe how I plan to achieve my two principal research goals: continue developing new measures of similarity, and improve the state-of-the-art in several areas involving sequential processing. With respect to the first goal, my specific objectives are: to propose data-driven learning schemes that integrate many-to-many correspondences and sensitivity to context and to design methods that incorporate linguistic knowledge and require no training data. With respect to the second goal, the areas in which I plan to make an impact are: letter-to-phoneme conversion, machine transliteration, statistical machine translation, and cognate identification. The development and application of word similarity methods leads to a number of challenging research problems, and promises to bring a unifying perspective on several distinct tasks that have so far been perceived as unrelated. Based on the experience of the last few years, I am convinced that this line of research will continue to have an impact beyond the areas of its original application.