Automating Dictionary Construction for Better Natural Language Processing

点击翻译按钮获取中文摘要

Language technology is an important component of many everyday software systems, such as predictive text entry on smartphones, spelling checkers in word processors, and automatic translation tools. Many language technology systems rely on dictionary-like resources that contain specialized information about words.Language is always changing. For example, new words, such as "mansplain" and "staycation", are coined every day. Established words regularly take on new meanings, such as the new senses of "post" and "wall" related to social media. New expressions also emerge, such as "flash mob". Dictionaries must therefore be continually updated to include these new usages.When information is missing from a language technology system's dictionary, the system won't work as well as it could otherwise. Keeping dictionaries up-to-date is therefore essential to building high quality language technology.Dictionaries are also important tools for language learners and translators, and are valuable in terms of cultural heritage. To be most useful for these purposes, dictionaries must also be kept up-to-date.Traditionally, lexicographers manually analyze large collections of texts to identify relevant facts about how words are used, and subsequently write dictionary entries. However, a staggering amount of text is written every day, particularly through online sources such as the web and social media. It is simply not possible for lexicographers to keep up with it all.How then, can dictionaries be kept up-to-date?Automatic methods are required. This research program will first consider how to best build massive document collections from the web. Drawing on recent linguistic theory and research in statistical machine learning, new natural language processing techniques will then be developed to analyze these documents to automatically "learn" the meanings of words and idiosyncratic expressions. New methods will then be developed to automatically discover how the meanings of words and expressions have changed. This research will have a particular focus on social media, because it contains an abundance of new words and usages.This research will enable the construction of higher quality dictionaries. This will have broad benefits for language technology systems, because they often rely on dictionaries. It will also benefit traditional dictionary users, including language learners and translators.