CROSS: Real-time Story Detection Across Multiple Massive Streams

CROSS：跨多个海量流的实时故事检测

• 批准号: EP/J020664/1
• 负责人: M Osborne
• 金额: $ 26.72万
• 依托单位: University of Edinburgh
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2012
• 资助国家: 英国
• 起止时间: 2012 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FJ020664%2F1
• 关键词: CROSS; Real; time; Story; Detection

The World is rapidly becoming more and more connected, with people communicating using multiple streams - Social Media, Newswire, Wikipedia etc - on a bewildering range of topics and at a furious rate. Twitter alone receives more than 250 million new posts every day (Tsotsis 2011). This massive interconnection means that content can appear and quickly spread through and across different streams. For example, in the recent London riots, many tweets reported the rioting events as they happened in real-time. However, not all content posted is either of good quality or is factually correct, complicating the job of monitoring such streams for any purpose. An example of this happened when a comedian spread false rumours on Twitter about Osama Bin Laden watching his television show (Lineham 2011). Communication streams are also known to spread rumours, outright misinformation and content with malicious intent. For instance, during the same riots, radicalising posts were spread calling for participation in the so-called "cyber-jihad" (BBC 2011). Systems that can identify such posts is of paramount importance for security monitoring purposes.On the other hand, not all information spread on mediums such as Twitter are accurate or interesting. This is compounded with the peculiarities of messages on modern social media (short, jargon, social context, etc.) where biased, incomplete, inaccurate and misleading messages are common. The latter makes it extremely challenging to automatically identify events worth monitoring for security purposes in real-time.We propose a distributed infrastructure to automatically identify important new events (aka stories) in real-time by combining and comparing multiple message streams. The value of such story detection to many applications is clearly increased the faster this can happen. A security agency using our system would be better prepared when dealing with fast moving events as they unfold. Indeed, in this project, the notion of importance will be defined within a security context. Given the fact that streams typically have possible bias and not everything present can be trusted, a key requirement of the system is minimising false positives (uninteresting stories that are discovered). Moreover, the effective management and efficient processing of multiple streams of real-time data poses new technological and scientific challenges:Challenge 1: Identify interesting new stories and not drown in a sea of false positives, yet reduce the effects of bias and rumour.Challenge 2: Minimise system latency, such that new stories are detected in real-time with low latency.We tackle the first challenge from the novel perspective of processing multiple streams and exploiting the fact that stories reported multiple times across several streams can cancel-out stream-specific bias and errors. For example, if a story is true, then it is more likely that it manifests in both Twitter and as an update to a Wikipedia article. Alternatively, a story might appear in Twitter and also appear in a governmental cable. The more often a story occurs within and across streams, the more likely it will be interesting. This is the cornerstone of our proposal, which we tackle by building upon modern first story detection techniques, adapted to account for bias and rumours.In the second challenge, we ensure low-latency story detection by using a distributed real-time data processing architecture (e.g. S4 or Storm), similar to MapReduce but better suited for real-time operations. Real-time architectures for dealing with massive-scale data are in their infancy, hence CROSS will present a first concrete application, with a corresponding development of best practices for such architectures.

世界正迅速变得越来越紧密，人们使用多种信息流-社交媒体，新闻网，维基百科等-就令人眼花缭乱的主题进行交流，并且速度非常快。仅Twitter每天就收到超过2.5亿条新帖子（Tsotsis 2011）。这种大规模的互联意味着内容可以通过不同的流出现并快速传播。例如，在最近的伦敦骚乱中，许多推文实时报道了骚乱事件。然而，并不是所有发布的内容都是高质量的或事实上是正确的，这使得出于任何目的监控此类流的工作变得复杂。这方面的一个例子是，一位喜剧演员在Twitter上散布关于奥萨马·本·拉登观看他的电视节目的谣言（Lineham 2011）。通信流也被称为传播谣言，彻头彻尾的错误信息和恶意内容。例如，在同一次骚乱中，传播了激进的帖子，呼吁参加所谓的“网络圣战”（BBC 2011年）。能够识别这些帖子的系统对于安全监控来说至关重要。另一方面，并非所有在Twitter等媒体上传播的信息都是准确或有趣的。这与现代社交媒体上信息的特殊性（简短，行话，社会背景等）相结合。在这些地方，偏见、不完整、不准确和误导性的信息很常见。后者使得它非常具有挑战性的自动识别事件值得监控的安全目的在real-time.We提出了一个分布式的基础设施，自动识别重要的新事件（又名故事）的实时组合和比较多个消息流。这种故事检测对许多应用程序的价值明显增加，这可以发生得越快。使用我们系统的安全机构在处理快速发展的事件时会做好更好的准备。事实上，在这个项目中，重要性的概念将在安全背景下定义。鉴于流通常具有可能的偏差并且并非所有存在的内容都可以信任的事实，系统的关键要求是最大限度地减少误报（发现的无趣故事）。此外，对多个实时数据流的有效管理和高效处理带来了新的技术和科学挑战：挑战1：识别有趣的新故事，而不是淹没在误报的海洋中，同时减少偏见和谣言的影响。挑战2：最小化系统延迟，使得新的故事在真实的中被检测到-我们从处理多个流的新颖角度来解决第一个挑战，并利用故事在多个流中多次报告的事实。流可以抵消流特定的偏差和误差。例如，如果一个故事是真实的，那么它更有可能同时出现在Twitter和维基百科文章的更新中。或者，一个故事可能出现在Twitter上，也可能出现在政府电报中。一个故事越经常发生在内部和跨流，它就越有可能有趣。这是我们方案的基石，我们通过构建现代的第一故事检测技术来解决这一问题，该技术适用于解释偏见和谣言。在第二个挑战中，我们通过使用分布式实时数据处理架构（例如S4或Storm）来确保低延迟的故事检测，该架构类似于MapReduce，但更适合实时操作。处理大规模数据的实时架构尚处于起步阶段，因此CROSS将提出第一个具体的应用程序，并相应地开发此类架构的最佳实践。

项目成果

Bieber no more : First Story Detection using Twitter and Wikipedia

• 发表时间: 2012
• 作者: M. Osborne;S. Petrovic;R. McCreadie;Craig Macdonald;I. Ounis;S. Petrovic

Can Twitter Replace Newswire for Breaking News?

• DOI: 10.1609/icwsm.v7i1.14450
• 发表时间: 2013-06
• 期刊: Proceedings of the International AAAI Conference on Web and Social Media
• 作者: S. Petrovic;M. Osborne;R. McCreadie;Craig Macdonald;I. Ounis;Luke Shrimpton

Incorporating Social Role Theory into Topic Models for Social Media Content Analysis

• DOI: 10.1109/tkde.2014.2359672
• 发表时间: 2015-04
• 期刊: IEEE Transactions on Knowledge and Data Engineering
• 影响因子: 8.9
• 作者: Wayne Xin Zhao;Jinpeng Wang;Yulan He;Jian-Yun Nie;Ji-Rong Wen;Xiaoming Li