Automatically building vocabularies from web forum text

每天都有数量惊人的文本被写在网上，其中大部分是在在线论坛上进行的讨论中 专注于赛车运动、宠物和收藏品等主题。这为企业提供了一个机会 分析这篇文章以深入了解消费者的需求和意见，从而为企业提供 竞争优势。由于数据量很大，对此类对话的分析最好是 自动完成。构建自动文本分析系统的关键是重要术语的词汇表。 这样的词汇可以从词典中获得；然而，由于在线的专门主题 论坛关注的焦点是，很多重要的词汇在一般词典中都没有列出。例如，在线论坛 包含许多不标准的单词，包括故意的变体拼写(例如“Cooolll”)和新奇 组合(如“deFuel”)，以及噪音(如用户名)和无意的拼写错误。 这个项目的目标是识别以上类别的单词，以便自动建立词汇表 在线论坛社区的关键术语。该项目将应用一系列统计技术，以绘制 关于包括词频、词中的字符序列和词所在的上下文的信息 出现。这样的词汇表的自动构建将有助于提高后续的性能 文本处理和分析工具，试图在给定的 在线论坛中存在大量杂乱无章的数据。反过来，这些工具可以提高 VerticalScope为论坛用户提供的在线服务，以及VerticalScope的吸引力 为各种企业提供服务。这些帮助VerticalScope成为数据科学研究和 发展，从而使加拿大经济受益。这将通过以下方式进一步惠及更广泛的公众 企业为了更好地了解消费者的需求，开发出更具吸引力的产品。

A staggering amount of text is written online every day, much of it in discussions taking place in online forums focused on topics such as motorsports, pets, and collectibles. This provides an opportunity for businesses to analyze this text to glean insights into consumer demands and opinions, which can then give a business a competitive advantage. Due to the large volume of data, the analyses of such conversations would ideally be done automatically. The key to building an automatic text analysis system is a vocabulary of important terms. Such a vocabulary could be taken from a dictionary; however, because of the specialized topics that online forums focus on, many important terms are not listed in general dictionaries. For example, online forums contain many non-standard words, including intentional variant spellings (e.g., "coooolll") and novel combinations (e.g., "defuel"), as well as noise (e.g., usernames) and unintentional spelling errors. The goal of this project is to identify the above categories of words in order to automatically build a vocabulary of key terms for an online forum community. The project will apply a range of statistical techniques that draw on information including word frequency, character sequences in words, and the contexts in which words appear. The automatic construction of such a vocabulary will help improve the performance of the subsequent text processing and analysis tools that try to zero in on relevant and important pieces of information given the vast amounts of noisy and unstructured data in online forums. In turn, such tools can improve the quality of the online services that VerticalScope provides to its forum users, as well as the attractiveness of VerticalScope's services to various businesses. These help VerticalScope become a leading player in data science research and development, hence benefiting the Canadian economy. This will further benefit the broader public by enabling businesses to better understand consumer needs to develop more-attractive products.