Processing queries on text streams

在过去的几年里，我们经历了用户生成内容（UGC）形式的文本信息爆炸。根据一些研究，网络日志（博客）账户数量级为1亿，参与前200个社交网络的用户接近13亿人，微博服务（如twitter、jaiku、friendfeed等）越来越受欢迎，留言板、评论网站和产品反馈论坛变得无处不在。社交媒体这个词被创造出来，指的是这种信息集合。所有这些应用程序的一个共同点是，个人产生的文本信息（和相关的多媒体内容）作为时间的函数。根据不同的应用程序的大小，风格，格式和复杂的文本内容产生的变化。在多伦多大学，我们一直在建立一个收集和分析社会媒体的系统。该系统从博客、新闻来源、社交网络和其他UGC论坛收集信息，从产生信息的个人中提取公开披露的信息（如年龄、地点、兴趣、职业等），并使其可供查询。该系统目前每天有数千人使用，可在互联网www.blogscope.net上查阅。它目前每天处理超过1000万份文档，并存储每天变化的8 TB数据。该提案将解决以促进信息发现的方式组织这一不断发展的信息集合的问题。我们将解决增加空间和时间维度的分面信息搜索问题，开发灵活的时间导航算法，并研究现代后端基础设施（如在map/reduce查询处理框架中运行的hadoop）中的查询处理和优化问题。

In the last couple of years we have been experiencing an explosion of text information in the form of user generated content (UGC). Web logs (blogs) account in the order of 100 millions according to some studies, user participation in the top 200 social networks is close to 1.3 billion people, microblogging services (such as twitter, jaiku, friendfeed, etc) are gaining popularity and message boards, review sites and product feedback forums are becoming ubiquitous. The term social media has been coined to refer to this information collective. A common denominator in all such applications is that individuals produce textual information (and associated multimedia content) as a function of time. Depending on the application the size, style, format and sophistication of text content produced varies. At the University of Toronto we have been building BlogScope a system for the collection and analysis of social media. The system collects information from blogs, news sources, social networks and other UGC forums, extracts information that is publicly disclosed from the individuals producing the information (such as age, location, interests, profession, etc) and makes it available for querying. The system is currently used by thousands of people daily and it is available on the internet at www.blogscope.net. It currently processes more than 10 million documents per day and warehouses 8TB of data that evolves daily. The proposal will address issues of organizing this constantly evolving collective of information in a ways that facilitate information discovery. We will address issues of faceted information search adding spatial and temporal dimensions, develop algorithms for flexible temporal navigation and research issues of query processing and optimization in modern back-end infrastructures such as hadoop operating in the map/reduce query processing framework.