CiC (RDDC): Wordsmith in the Cloud - Refining Language Models Using Web-Scale Language Networks

对于网络用户来说，在网络社区中表达自己的想法从未像现在这样容易。在线对话产生的文本信息过载引起了各种消费群体的关注。许多管理这种信息过载的有效工具，如文本检索系统，都依赖于统计语言模型的使用。然而，语言模型的质量受到数据稀疏性、上下文不匹配以及无法对语义关系进行建模的限制。在这个项目中，将使用Azure和Hadoop等云计算系统和新颖的分布式算法来从Web规模的文本集合中提取异质语言网络。这些语言网络将被用来使不同领域的语言模型更平滑和更具情境化，从而使其准确和健壮。改进的语言模型将有助于改进最先进的文本检索和挖掘技术，增强真实用户跨越社区和语言边界的信息获取和知识获取体验。这些技术和资源(例如，语言网络和改进的语言模型)将使分析社交媒体和许多其他领域中的文本内容的广泛用户受益。该项目的研究成果，包括软件工具和资源，将在项目网站(http://www-personal.umich.edu/~qmei/wordsmith/).上公布

It has never been easier for Web users to express themselves in online communities. The overload of text information generated from online conversations is of concern to various consumer groups. Many effective tools to manage this information overload, such as text retrieval systems, rely on the use of statistical language models. The quality of language models is however limited by the sparseness of data, the mismatch of context, and the incapability of modeling semantic relations. In this project, cloud computing systems like Azure and Hadoop and novel distributed algorithms will be employed to extract heterogeneous language networks from Web-scale text collections. These language networks will be used to smooth and contextualize language models in various domains, making them accurate and robust. The refined language models will help improve state-of-the-art text retrieval and mining techniques, enhancing the information access and knowledge acquisition experience of real users across community and language boundaries. The techniques and resources (e.g., language networks and refined language models) will benefit a broad range of users that analyze text content in social media and many other domains. Research results of this project, including software tools and resources, will be published on the project Web site (http://www-personal.umich.edu/~qmei/wordsmith/).