CiC (RDDC): Wordsmith in the Cloud - Refining Language Models Using Web-Scale Language Networks

网络用户在网络社区中表达自己从未像现在这样容易。在线对话产生的文本信息过载是各种消费群体关注的问题。管理这种信息过载的许多有效工具，如文本检索系统，都依赖于统计语言模型的使用。然而，语言模型的质量受到数据稀疏性、上下文不匹配以及无法对语义关系建模的限制。在这个项目中，将使用Azure和Hadoop等云计算系统以及新颖的分布式算法从web规模的文本集合中提取异构语言网络。这些语言网络将用于平滑和上下文化各个领域的语言模型，使其准确和健壮。精细化的语言模型将有助于改进最先进的文本检索和挖掘技术，增强跨社区和语言边界的真实用户的信息访问和知识获取体验。这些技术和资源（例如，语言网络和精炼的语言模型）将使分析社交媒体和许多其他领域文本内容的广泛用户受益。该项目的研究成果，包括软件工具和资源，将在该项目的网站（http://www-personal.umich.edu/~qmei/wordsmith/）上公布。

It has never been easier for Web users to express themselves in online communities. The overload of text information generated from online conversations is of concern to various consumer groups. Many effective tools to manage this information overload, such as text retrieval systems, rely on the use of statistical language models. The quality of language models is however limited by the sparseness of data, the mismatch of context, and the incapability of modeling semantic relations. In this project, cloud computing systems like Azure and Hadoop and novel distributed algorithms will be employed to extract heterogeneous language networks from Web-scale text collections. These language networks will be used to smooth and contextualize language models in various domains, making them accurate and robust. The refined language models will help improve state-of-the-art text retrieval and mining techniques, enhancing the information access and knowledge acquisition experience of real users across community and language boundaries. The techniques and resources (e.g., language networks and refined language models) will benefit a broad range of users that analyze text content in social media and many other domains. Research results of this project, including software tools and resources, will be published on the project Web site (http://www-personal.umich.edu/~qmei/wordsmith/).