RAnDMS (Real time Analysis of Digital Media Streams)

RAnDMS将研究，实施和评估实时数据和可视化分析技术，使情报机构，国防部，警察和应急响应人员能够使用社交和传统媒体流的网络规模数据来监控和了解当地，区域和全球事件。情报收集任务将被定义为确定、关联、整合和提供数据和信息，以便在出现情况时了解情况。目前的技术并没有提供高效和有效的解决方案，因为它主要集中在检测关键字和标签使用的趋势。虽然这能够发现数据中的总体模式，但它只能检索相关文档，而不能对所包含的信息进行任何关联和集成。此外，有关当地情况和事件的信息可能只在少数文件中讨论，被忽略了。在RAnDMS中，数据分析将侧重于从媒体流中捕获信息;阐明从全球到地方的所有层面的情况。这些资料将有助于情报界的决策，预期这将提高他们监测与国土安全和维持和平努力有关的事件和局势的能力。科学上的挑战是，这些数据流中的数据和信息：（一）数量大，而且不断增加;（二）经常重复、不完整、不精确和不正确;（三）以非正式的风格编写（即简短、未经编辑和对话式）;（四）一般涉及短期的时代精神。这些特点使得分析非常困难，特别是考虑到情报界的主要要求是：（i）文件必须实时处理;（ii）相关信息可能处于分布的长尾中，即可能很少被提及。我们将提供高效和有效的技术，能够将每个文档与其上下文相关联。文档上下文由四个维度提供：（谁）文档的作者，（何时）发送时间，（何处）文档中提到的位置以及（什么）具有类似内容的其他文档。这些信息要么由媒体流提供，要么使用高效的统计文本挖掘技术从文档内容中提取。通过在这四个维度方面解释文档，我们能够：（i）检测事件，即文档及其内容（什么）围绕时间和地点聚集;（ii）从内容中分析作者（什么和在哪里）他们创建的文件;以及（iii）确定文档中缺失或模糊的信息，可视化分析将通过提供多个视图来促进对信息的探索;在四个维度上实现有针对性的调查和趋势可视化。我们将设计的方法，（一）建议正确的详细程度（粒度）的用户在快速变化的环境中的重点;（二）提醒用户的任何重大发展以外的他们目前的观点;和（三）使用户能够理解如何通过浏览沿着所有信息沿着时间维度的当前状态的形成。这些方法将能够看穿社交媒体中经常围绕事件的不相关的玩笑（噪音），并直接找到隐藏在分布长尾中的相关信息。RAnDMS将在2012/13年期间发生的相关事件期间进行测试，以执行支持情报操作人员的任务。我们将在国际期刊和会议上发表研究及其发现。根据国防部的协议，我们还将通过生成一个公开可用的任务（包括语料库，资源等）来创建公共研究资源。以便与其他研究人员的研究结果进行比较。

RAnDMS will study, implement and evaluate Real-time Data and Visual Analytic techniques to enable intelligence agencies, the MoD, the police and emergency responders to monitor and make sense of local, regional and global events using web-scale data from social and traditional media streams. The intelligence gathering task will be defined as identifying, correlating, integrating and presenting data and information, in order to understand situations as they arise. Current technology does not provide efficient and effective solutions, as it mainly focuses on detecting trends in the use of keywords and tags. While this is able to spot overall patterns in the data, it just enables the retrieval of relevant documents, without any correlation and integration of the contained information. Moreover, information concerning local situations and events, which may only be discussed within a handful of documents, is ignored.Within RAnDMS data analytics will focus on enabling the capture of information from media streams; illuminating situations at all levels, from global to local. This information will support decision making for the intelligence community, which is expected to increase their ability to monitor events and situations relevant to homeland security and to peace-keeping efforts. The scientific challenge is that data and information in these streams are: (i) high in volume, and constantly increasing, (ii) often duplicated, incomplete, imprecise and incorrect; (iii) written in informal style (i.e. short, unedited and conversational); and (iv) generally concerning the short-term zeitgeist. These characteristics make analysis very hard, especially when considering that major requirements of the intelligence community are that (i) documents must be processed in real-time and (ii) the relevant information may be in the long-tail of the distribution, i.e. it may be mentioned very infrequently. We will provide highly efficient and effective technologies able to associate each document with its context. A documents context is provided by four dimensions: (who) the author of the document, (when) the time it was sent, (where) the location referred to in the document and (what) other documents with similar content. This information is either provided by the media stream or extracted from the document's content using efficient statistical text-mining techniques. By interpreting documents in terms of these four dimensions we enable: (i) the detection of events, i.e. documents and their content (what) are clustered around a time and place; (ii) the profiling of authors from the content (what and where) of the documents they have created; and (iii) determine information that is missing or ambiguous in document, using information present in the documents within their context.Visual analytics will facilitate the exploration of the information by providing multiple views; enabling focused investigation and trend visualisations across the four dimensions. We will devise methods to (i) suggest the right level of detail (granularity) for the user focus in rapidly changing environments; (ii) alert users to any significant development outside of their current viewpoint; and (iii) enable users to understand how the current state of affairs came into being by browsing along the all information along the time dimension. Methods will en able to see through the irrelevant banter (noise) that often surround events in social media and go directly to the relevant information that can be hidden in the long tail of the distribution. RAnDMS will be tested on the task of supporting intelligence operators during relevant events happening during 2012/13. We will publish the research and its findings in international journal and conferences. Subject to MoD agreement, we will also create public research resources by generating one publicly available task (inclusive of corpora, resources, etc.) to enable comparison of research results by other researchers.