CAREER: Detecting Structured Anomalies in Large-Scale Sequential Decision Problems and Latent Variable Models

本研究项目将为异常检测这一历史悠久、应用广泛的课题开发新的统计方法和理论。例如，制造业中的故障检测、公共卫生中的疾病爆发检测、信号处理中的频谱感知、教育测试中的项目变化检测以及电子商务中的欺诈检测。虽然传统方法主要专注于独立识别偏离正常状态的数据点，但大数据时代出现了新的挑战，因为异常往往涉及具有复杂结构的海量数据。该项目将开发新的统计方法和理论结果以及计算工具，以系统地处理大规模数据中结构化异常的检测。除了技术贡献外，该项目开发的方法还将对其他学科的研究产生积极影响。例如，本项目开发的变化检测方法将有助于对教育测试中的题库质量进行监控，以提高测试的效度和可靠性。该项目还将实施一项教育计划，其中包括让研究生和本科生参与研究活动，创建新的课程，并与教育机构进行接触。该项目的成果将通过期刊出版物和会议广泛传播，并将开发可供公众使用的统计软件。具体地说，本项目将重点研究大规模序贯决策和潜变量模型中的两类问题。第一类问题涉及大规模流数据，由于数据采集技术的快速发展，这在最近几年变得普遍。该项目将建立一个通用的复合序贯决策理论框架，以量化大规模在线变化检测问题的程序性能，并在该框架下制定有效的序贯决策。第二类问题是具有结构异常值的高维广义潜在因子模型。该项目将开发有效的模型估计和统计推断方法，并为其准确性和可靠性提供理论保证。将讨论模型的可辨识性和可估计性等基本问题。此外，将开发新的技术工具，以解决上述问题中的理论和方法挑战。例如，将开发一种用于生活在非欧几里德空间上的随机过程的单调耦合技术，以增强对多数据流问题的顺序决策的理解。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will develop new statistical methods and theory for anomaly detection, which is a topic with a long history and wide-ranging applications. Examples include fault detection in manufacturing, disease outbreak detection in public health, spectrum sensing in signal processing, item change detection in educational testing, and fraud detection in e-commerce. While traditional methods mainly focus on identifying data points deviating from their normal states independently, new challenges arise in the big-data era as the anomalies often involve massive data with complex structures. This project will develop novel statistical methods and theoretical results along with computational tools to systematically deal with the detection of structured anomalies in large-scale data. In addition to the technical contribution, the methods developed in this project will positively impact research in other disciplines. For instance, the change detection method developed in this project will aid the monitoring of item pool quality in educational testing to improve the validity and reliability of the tests. This project will also implement an educational plan which includes engaging graduate and undergraduate students in research activities, creating a new curriculum, and outreach to educational institutes. The outcome of the project will be broadly disseminated through journal publications and conferences, and publicly available statistical software will be developed. Specifically, this project will focus on two classes of problems in large-scale sequential decision-making and latent variable models. The first class of problems involves large-scale streaming data, which have become common in recent years, owing to the rapid development in data acquisition technologies. The project will establish a general compound sequential decision theory framework to quantify the performance of procedures for large-scale online change detection problems and develop efficient sequential decisions under this framework. The second class of problems is on high-dimensional generalized latent factor models with structured outliers. The project will develop efficient model estimation and statistical inference methods and provide theoretical guarantees on their accuracy and reliability. Fundamental issues such as identifiability and estimability of the model will be addressed. Moreover, novel technical tools will be developed to address theoretical and methodological challenges in the above problems. For example, a monotone coupling technique for stochastic processes living on a non-Euclidean space will be developed to enhance the understanding of sequential decisions for multi-stream problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.