Processing queries on text streams

在过去的几年里，我们经历了用户生成内容(UGC)形式的文本信息的爆炸性增长。根据一些研究，前200名社交网络的用户参与度接近13亿人，微博服务(如Twitter、Jaiku、FriendFeed等)越来越受欢迎，留言板、评论网站和产品反馈论坛变得无处不在。社交媒体这一术语被创造出来，指的是这种信息集合。所有这类应用中的一个共同点是，个人产生文本信息(以及相关的多媒体内容)作为时间的函数。根据应用程序的不同，生成的文本内容的大小、样式、格式和复杂程度也不同。在多伦多大学，我们一直在建立一个用于收集和分析社交媒体的BlogScope系统。该系统从博客、新闻来源、社交网络和其他UGC论坛收集信息，从产生信息的个人那里提取公开披露的信息(如年龄、地点、兴趣、职业等)，并提供查询。该系统目前每天有数以千计的人在使用，它可以在互联网上获得，网址是www.blogscope e.net。它目前每天处理1000多万份文档，并存储每天发展的8TB数据。该提案将解决以促进信息发现的方式组织这一不断演变的信息集合的问题。我们将解决增加空间和时间维度的分面信息搜索问题，开发灵活的时间导航算法，并研究现代后端基础设施中的查询处理和优化问题，例如在MAP/REDUE查询处理框架中操作的Hadoop。

In the last couple of years we have been experiencing an explosion of text information in the form of user generated content (UGC). Web logs (blogs) account in the order of 100 millions according to some studies, user participation in the top 200 social networks is close to 1.3 billion people, microblogging services (such as twitter, jaiku, friendfeed, etc) are gaining popularity and message boards, review sites and product feedback forums are becoming ubiquitous. The term social media has been coined to refer to this information collective. A common denominator in all such applications is that individuals produce textual information (and associated multimedia content) as a function of time. Depending on the application the size, style, format and sophistication of text content produced varies. At the University of Toronto we have been building BlogScope a system for the collection and analysis of social media. The system collects information from blogs, news sources, social networks and other UGC forums, extracts information that is publicly disclosed from the individuals producing the information (such as age, location, interests, profession, etc) and makes it available for querying. The system is currently used by thousands of people daily and it is available on the internet at www.blogscope.net. It currently processes more than 10 million documents per day and warehouses 8TB of data that evolves daily. The proposal will address issues of organizing this constantly evolving collective of information in a ways that facilitate information discovery. We will address issues of faceted information search adding spatial and temporal dimensions, develop algorithms for flexible temporal navigation and research issues of query processing and optimization in modern back-end infrastructures such as hadoop operating in the map/reduce query processing framework.