Processing queries on text streams

在过去的几年里，我们经历了用户生成内容（UGC）形式的文本信息的爆炸式增长。据一些研究显示，网络日志（博客）账户约为1亿，前200名社交网络的用户参与人数接近13亿人，微博服务（如twitter、jaiku、friendfeed等）越来越受欢迎，留言板、评论网站和产品反馈论坛无处不在。社交媒体这个词就是用来指代这种信息集合体的。所有这类应用程序的一个共同点是，个人生成文本信息（以及相关的多媒体内容）作为时间的函数。根据应用程序的不同，生成的文本内容的大小、样式、格式和复杂程度各不相同。在多伦多大学，我们一直在建立BlogScope，一个收集和分析社会媒体的系统。该系统从博客、新闻来源、社交网络和其他UGC论坛收集资料，并从提供资料的个人那里提取公开披露的资料（例如年龄、地点、兴趣、职业等），供市民查询。该系统目前每天有数千人使用，可在互联网上访问www.blogscope.net。它目前每天处理超过1000万份文档，每天存储8TB的数据。该提案将解决以促进信息发现的方式组织这种不断发展的信息集体的问题。我们将解决多面信息搜索增加空间和时间维度的问题，开发灵活的时间导航算法，研究现代后端基础设施（如在map/reduce查询处理框架中运行的hadoop）的查询处理和优化问题。

In the last couple of years we have been experiencing an explosion of text information in the form of user generated content (UGC). Web logs (blogs) account in the order of 100 millions according to some studies, user participation in the top 200 social networks is close to 1.3 billion people, microblogging services (such as twitter, jaiku, friendfeed, etc) are gaining popularity and message boards, review sites and product feedback forums are becoming ubiquitous. The term social media has been coined to refer to this information collective. A common denominator in all such applications is that individuals produce textual information (and associated multimedia content) as a function of time. Depending on the application the size, style, format and sophistication of text content produced varies. At the University of Toronto we have been building BlogScope a system for the collection and analysis of social media. The system collects information from blogs, news sources, social networks and other UGC forums, extracts information that is publicly disclosed from the individuals producing the information (such as age, location, interests, profession, etc) and makes it available for querying. The system is currently used by thousands of people daily and it is available on the internet at www.blogscope.net. It currently processes more than 10 million documents per day and warehouses 8TB of data that evolves daily. The proposal will address issues of organizing this constantly evolving collective of information in a ways that facilitate information discovery. We will address issues of faceted information search adding spatial and temporal dimensions, develop algorithms for flexible temporal navigation and research issues of query processing and optimization in modern back-end infrastructures such as hadoop operating in the map/reduce query processing framework.