Adaptive and Scalable Event Detection Techniques for Twitter Data Streams

Twitter拥有2.71亿月活跃用户，每天发布超过5亿条推文，是目前最受欢迎、增长最快的微博服务。因此，Twitter越来越多地被用作时事信息的来源。对于传统媒体，如报纸档案和新闻网站，事件检测问题已经从话题检测和跟踪（TDT）领域得到了解决。然而，Twitter数据流中的主题检测带来了一系列额外的挑战。首先，由于篇幅限制，Twitter文档比传统新闻文章要短得多，因此更难分类。其次，推文没有经过编辑，因此可能包含大量的垃圾邮件、错别字、俚语等。最后，推文的生成速度非常快，并且随着将来越来越多的用户采用Twitter，它将继续增加。已经提出了几种用于社交媒体（特别是Twitter）事件检测的方法。然而，这些建议中的大多数倾向于只关注信息提取方面，而经常忽略输入的流性质。例如，许多技术都带有一组复杂但固定的参数，用于控制检测哪些事件。假设这些参数是通过在样本数据集上运行算法来经验地确定的，直到它产生期望的结果。我们认为，这种方法既不现实也不可行的原因有几个。首先，流中的数据可能会发生质的变化，这可能需要参数来适应，以便继续准确地检测事件。其次，这些参数不仅控制了技术的基于任务的性能，而且还控制了运行时的性能。因此，使用固定参数可以防止这些方法随流中的定量变化进行缩放。在这个项目中，我们建议在数据流管理系统（DSMS）研究的传统中解决Twitter中自适应和可扩展事件检测的需求。为了使项目更加集中，我们将专注于第一故事检测的具体任务，即一般（未知）事件的检测，这被定义为TDT的子任务之一。我们计划在三个单独的工作包中解决这些问题。在第一个工作包中，我们将研究事件检测方法如何适应流的内容，方法是在处理流之前探索更好的方法来分割流，并在处理过程中调整方法参数。第二个工作包将解决可伸缩性需求，既可以随一个流的容量上下扩展，也可以扩展到多个并行流。最后，第三个工作包将专门用于评估事件检测技术的重要任务。

With 271 million monthly active users that produce over 500 million tweets per day, Twitter is currently the most popular and fastest-growing microblogging service. Twitter is therefore increasingly used as a source of information on current events as they unfold.For traditional media such as newspaper archives and news website, the problem of event detection has been addressed by research from the area of Topic Detection and Tracking (TDT). However, topic detection in Twitter data streams raises a set of additional challenges. First, Twitter documents are much shorter than traditional news articles due to their length limitation and therefore harder to classify. Second, tweets are not edited and can therefore contain a substantial amount of spam, typos, slang, etc. Finally, the rate with which tweets are being produced is very bursty and will continue to increase as more users adopt Twitter in the future.Several approaches for event detection in social media and, in particular, for Twitter have been proposed. However, most of these proposals tend to focus exclusively on the information extraction aspect and often ignore the streaming nature of the input. For example, many techniques come with a complex but fixed set of parameters that control which events are detected. It is assumed that these parameters are empirically determined by running the algorithm on a sample data set until it produces the desired result. We argue that there are several reasons why this approach is neither realistic nor feasible. First, the data in the stream may undergo qualitative changes that may require parameters to adapt in order to continue to detect events accurately. Second, these parameter not only control the task-based performance of a technique but also the run-time performance. Working with fixed parameters therefore prevents these approaches to scale with quantitative changes in the stream.In this project, we propose to address the need for adaptive and scalable event detection in Twitter in the tradition of Data Stream Management Systems (DSMS) research. In order to focus the project, we will concentrate on the specific task of first story detection, i.e., the detection of general (unknown) events, which is defined as one of the subtasks of TDT. We plan to address these issues in three separate work packages. In the first work package, we will study how event detection methods can adapt to the content of the stream by exploring better ways to segment the stream before it is processed and by adjusting method parameters during processing. The second work package will address scalability requirements in terms of scaling up and down with the volume of one stream but also in terms of scaling up to several parallel streams. Finally, a third work package will be dedicated to the non-trivial task of evaluating event detection techniques.