CAREER: Detecting Structured Anomalies in Large-Scale Sequential Decision Problems and Latent Variable Models

点击翻译按钮获取中文摘要

This research project will develop new statistical methods and theory for anomaly detection, which is a topic with a long history and wide-ranging applications. Examples include fault detection in manufacturing, disease outbreak detection in public health, spectrum sensing in signal processing, item change detection in educational testing, and fraud detection in e-commerce. While traditional methods mainly focus on identifying data points deviating from their normal states independently, new challenges arise in the big-data era as the anomalies often involve massive data with complex structures. This project will develop novel statistical methods and theoretical results along with computational tools to systematically deal with the detection of structured anomalies in large-scale data. In addition to the technical contribution, the methods developed in this project will positively impact research in other disciplines. For instance, the change detection method developed in this project will aid the monitoring of item pool quality in educational testing to improve the validity and reliability of the tests. This project will also implement an educational plan which includes engaging graduate and undergraduate students in research activities, creating a new curriculum, and outreach to educational institutes. The outcome of the project will be broadly disseminated through journal publications and conferences, and publicly available statistical software will be developed. Specifically, this project will focus on two classes of problems in large-scale sequential decision-making and latent variable models. The first class of problems involves large-scale streaming data, which have become common in recent years, owing to the rapid development in data acquisition technologies. The project will establish a general compound sequential decision theory framework to quantify the performance of procedures for large-scale online change detection problems and develop efficient sequential decisions under this framework. The second class of problems is on high-dimensional generalized latent factor models with structured outliers. The project will develop efficient model estimation and statistical inference methods and provide theoretical guarantees on their accuracy and reliability. Fundamental issues such as identifiability and estimability of the model will be addressed. Moreover, novel technical tools will be developed to address theoretical and methodological challenges in the above problems. For example, a monotone coupling technique for stochastic processes living on a non-Euclidean space will be developed to enhance the understanding of sequential decisions for multi-stream problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.