Automatically building vocabularies from web forum text

点击翻译按钮获取中文摘要

A staggering amount of text is written online every day, much of it in discussions taking place in online forums focused on topics such as motorsports, pets, and collectibles. This provides an opportunity for businesses to analyze this text to glean insights into consumer demands and opinions, which can then give a business a competitive advantage. Due to the large volume of data, the analyses of such conversations would ideally be done automatically. The key to building an automatic text analysis system is a vocabulary of important terms. Such a vocabulary could be taken from a dictionary; however, because of the specialized topics that online forums focus on, many important terms are not listed in general dictionaries. For example, online forums contain many non-standard words, including intentional variant spellings (e.g., "coooolll") and novel combinations (e.g., "defuel"), as well as noise (e.g., usernames) and unintentional spelling errors. The goal of this project is to identify the above categories of words in order to automatically build a vocabulary of key terms for an online forum community. The project will apply a range of statistical techniques that draw on information including word frequency, character sequences in words, and the contexts in which words appear. The automatic construction of such a vocabulary will help improve the performance of the subsequent text processing and analysis tools that try to zero in on relevant and important pieces of information given the vast amounts of noisy and unstructured data in online forums. In turn, such tools can improve the quality of the online services that VerticalScope provides to its forum users, as well as the attractiveness of VerticalScope's services to various businesses. These help VerticalScope become a leading player in data science research and development, hence benefiting the Canadian economy. This will further benefit the broader public by enabling businesses to better understand consumer needs to develop more-attractive products.