ATD: Scanning Dynamic Spatial-Temporal Discrete Events for Threat Detection

点击翻译按钮获取中文摘要

The overarching research objective of this project is to develop a statistical framework for detecting anomalies from spatial-temporal discrete event data. Nowadays, a large volume of such event data dispersed over space and time are becoming increasingly available in a wide variety of applications, such as human activity data, social network data, and crime data. The observations of the discrete events can occur in continuous time and locations, and there can be a complex text description of such events. The discrete event data contain rich correlation and causal information, which can potentially be used to infer the dynamics of the underlying systems and detect threats. The project aims to develop statistical methods to harvest this potential in threat detection using discrete events and address the algorithmic and computational challenges. The developed methods will go beyond the status-quo model estimation by considering more general statistical inference problems such as hypothesis tests and likelihood-based inference. The developed methods are general and can be used for various discrete event data. The project will specifically demonstrate the effectiveness of the developed methods on a large-scale crime dataset collected by the Atlanta Police Department. Recently, point process models have been proven an effective model for capturing the correlation structure in discrete events. While much success has been achieved in estimating the self-exciting spatial-temporal point process models, it remains unclear how one can perform anomaly detection leveraging these models, since (1) detection (which can be cast as hypothesis test) is inherently different from estimation, which involves different kinds of statistics and performance metrics; (2) in various situations, there is a large number of discrete events over broad spatial areas, and the goal is to detect a small cluster of related events, which amounts to "finding a needle in a haystack", thus there is a need to develop powerful and computationally efficient statistics; (3) the normal or reference state can be complex and dynamic and methods need to adapt to the slowly time-varying normal state. The project will address these challenges and provide answers to two related fundamental questions: how to detect clusters of correlated events from a large amount of data using the point process model, and how to estimate time-varying background normal pattern. The proposed work will advance the state-of-art for scan statistic research and build a novel connection between pseudo-likelihood estimation and reinforcement learning. The developed methods will be tested in a specific application of crime data analysis. The proposed education activities will involve students at all levels in rigorous mathematical and statistical training and gain hands-on data analysis skills.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.