RAnDMS (Real time Analysis of Digital Media Streams)

RAnDMS将研究、实施和评估实时数据和可视化分析技术，使情报机构、国防部、警察和应急响应人员能够使用来自社会和传统媒体流的网络规模数据来监控和理解本地、区域和全球事件。情报收集任务将被定义为识别、关联、整合和呈现数据和信息，以便了解出现的情况。目前的技术并没有提供高效的解决方案，因为它主要集中在检测关键字和标签的使用趋势。虽然这能够发现数据中的整体模式，但它只支持检索相关文档，而不需要对所包含的信息进行任何关联和集成。此外，有关当地情况和事件的信息可能只在少数文件中讨论，却被忽略了。在RAnDMS中，数据分析将侧重于从媒体流中捕获信息；照亮了从全球到地方的各个层面的情况。这些资料将支持情报界的决策，预期情报界将提高其监测与国土安全和维持和平努力有关的事件和局势的能力。科学上的挑战是，这些流中的数据和信息：(i)量大，而且不断增加；（ii）经常重复、不完整、不精确和不正确；（iii）以非正式风格写作（即简短，未经编辑和会话）；（四）一般涉及短期的时代精神。这些特点使分析变得非常困难，特别是考虑到情报界的主要要求是(i)文件必须实时处理，（ii）相关信息可能处于分发的长尾，即可能很少被提及。我们将提供能够将每个文档与其上下文关联起来的高效和有效的技术。文档上下文由四个维度提供：（谁）文档的作者，（何时）发送的时间，（何处）文档中引用的位置，以及（什么）具有类似内容的其他文档。这些信息要么由媒体流提供，要么使用高效的统计文本挖掘技术从文档内容中提取。通过从这四个维度解释文档，我们可以：(i)检测事件，即文档及其内容（什么）围绕时间和地点聚集；（ii）根据作者所创建文件的内容（内容和地点）对其进行分析；（iii）利用文件中在其上下文中存在的信息，确定文件中缺失或模糊的信息。可视化分析通过提供多种视图，将有助于对信息的探索；实现跨四个维度的重点调查和趋势可视化。我们将设计方法来(i)建议用户在快速变化的环境中关注的正确细节级别（粒度）；（ii）提醒使用者注意他们目前观点以外的任何重大发展；（iii）通过时间维度上的所有信息浏览，让用户了解当前事件是如何产生的。方法将能够看穿社交媒体中经常围绕事件的无关玩笑（噪音），并直接找到可能隐藏在分布长尾中的相关信息。RAnDMS将在2012/13年度的相关事件中测试支持情报操作员的任务。我们将在国际期刊和会议上发表研究结果。根据国防部的协议，我们还将通过生成一个公开可用的任务（包括语料库、资源等）来创建公共研究资源，以便其他研究人员可以比较研究结果。

RAnDMS will study, implement and evaluate Real-time Data and Visual Analytic techniques to enable intelligence agencies, the MoD, the police and emergency responders to monitor and make sense of local, regional and global events using web-scale data from social and traditional media streams. The intelligence gathering task will be defined as identifying, correlating, integrating and presenting data and information, in order to understand situations as they arise. Current technology does not provide efficient and effective solutions, as it mainly focuses on detecting trends in the use of keywords and tags. While this is able to spot overall patterns in the data, it just enables the retrieval of relevant documents, without any correlation and integration of the contained information. Moreover, information concerning local situations and events, which may only be discussed within a handful of documents, is ignored.Within RAnDMS data analytics will focus on enabling the capture of information from media streams; illuminating situations at all levels, from global to local. This information will support decision making for the intelligence community, which is expected to increase their ability to monitor events and situations relevant to homeland security and to peace-keeping efforts. The scientific challenge is that data and information in these streams are: (i) high in volume, and constantly increasing, (ii) often duplicated, incomplete, imprecise and incorrect; (iii) written in informal style (i.e. short, unedited and conversational); and (iv) generally concerning the short-term zeitgeist. These characteristics make analysis very hard, especially when considering that major requirements of the intelligence community are that (i) documents must be processed in real-time and (ii) the relevant information may be in the long-tail of the distribution, i.e. it may be mentioned very infrequently. We will provide highly efficient and effective technologies able to associate each document with its context. A documents context is provided by four dimensions: (who) the author of the document, (when) the time it was sent, (where) the location referred to in the document and (what) other documents with similar content. This information is either provided by the media stream or extracted from the document's content using efficient statistical text-mining techniques. By interpreting documents in terms of these four dimensions we enable: (i) the detection of events, i.e. documents and their content (what) are clustered around a time and place; (ii) the profiling of authors from the content (what and where) of the documents they have created; and (iii) determine information that is missing or ambiguous in document, using information present in the documents within their context.Visual analytics will facilitate the exploration of the information by providing multiple views; enabling focused investigation and trend visualisations across the four dimensions. We will devise methods to (i) suggest the right level of detail (granularity) for the user focus in rapidly changing environments; (ii) alert users to any significant development outside of their current viewpoint; and (iii) enable users to understand how the current state of affairs came into being by browsing along the all information along the time dimension. Methods will en able to see through the irrelevant banter (noise) that often surround events in social media and go directly to the relevant information that can be hidden in the long tail of the distribution. RAnDMS will be tested on the task of supporting intelligence operators during relevant events happening during 2012/13. We will publish the research and its findings in international journal and conferences. Subject to MoD agreement, we will also create public research resources by generating one publicly available task (inclusive of corpora, resources, etc.) to enable comparison of research results by other researchers.