ATD: Collaborative Research: Efficient sampling for real-time detection and isolation of threats in networks

虽然现代技术产生了多个相互关联的数据源，这些数据源可以几乎实时地观察到，但物理约束和预算限制要求对这些数据流进行有效的采样。将这些限制因素纳入威胁检测系统的设计中，有可能大大节省资源。该项目的目标是发展基本的统计理论和方法，以便对网络进行高效、实时的采样，并随后检测、识别和预防不同性质的威胁，如恐怖活动和信贷欺诈。开发的方法将在现实世界的数据上进行测试，在那里，危机时期的潜在社区结构将通过手机通话记录被发现。在这个项目中获得的理论知识将被纳入研究生水平课程的材料中，包括适应性实验设计和顺序检测。两名研究生将在本研究中做出重要贡献。该项目将尽一切努力让代表性不足群体的合格学生参与这些研究活动。本研究将解决两个基本研究问题：1)如何实时检测和识别受采样约束的网络数据中的异常簇；2)如何有效地分配有限的资源，以延迟或阻止已识别的异常簇的威胁实现。这些问题的数学公式导致了基于网络、自适应实验设计和顺序检测的新问题，其解决方案需要创造性地结合各个领域的工具，如统计推断、顺序分析和信息论。在这项工作中发展的理论和方法将指导威胁检测算法的发展，并将在具体应用中进行测试，例如在危机时期的社交网络，将根据手机通话数据发现。总的来说，这是一个多学科的建议，跨越社会科学，统计学和工程学，其目标是获得一个有效的网络采样方案和新的威胁检测算法库，基于强大的理论背景。

While modern technologies generate multiple, interconnected sources of data, which can be observed in nearly real time, physical constraints and budget limitations require efficient sampling of these data streams. Incorporating such constraints in the design of threat detection systems has the potential to lead to significant savings in resources. The goal of this project is to develop fundamental statistical theory and methods for the efficient, real-time sampling of networks, and the subsequent detection, identification and prevention of threats of different nature, such as terrorist activities and credit fraud. The developed methodologies will be tested on real-world data, where the underlying community structure in times of crisis will be discovered using cell-phone call records. The theoretical knowledge that will be gained in this project will be incorporated into the material of graduate-level courses that cover adaptive experimental design and sequential detection. Two graduate students will contribute significantly in this research. The project will make every effort to include qualified students of underrepresented groups in these research activities.This research will address two fundamental research questions: 1) how to detect and identify, in real time, anomalous clusters in network data subject to sampling constraints, and 2) how to efficiently allocate limited resources in order to delay or prevent the realization of threats from the identified anomalous clusters. The mathematical formulation of these questions leads to novel problems in network-based, adaptive experimental design and sequential detection, whose solutions require the creative combination of tools from various fields, such as statistical inference, sequential analysis, and information theory. The theory and methods developed in this work will guide the development of threat detection algorithms and will be tested in concrete applications, such as social networks in times of crisis that will be discovered based on cell-phone call data. Overall, this is a multidisciplinary proposal, spanning social sciences, statistics, and engineering, whose goal is to obtain an arsenal of efficient network sampling schemes and novel threat detection algorithms, grounded on a strong theoretical background.