CAREER: Simultaneous and Sequential Inference of High-dimensional Data with Sparse Structure

在海量复杂数据中准确可靠地恢复稀疏信号一直是许多科学领域的基本问题。发现过程通常涉及通过大量假设进行广泛筛选，以分离感兴趣的信号并识别其模式。这种情况可以用大海捞针来形容。尽管在数据筛选、模式识别和相关领域的方法学工作取得了巨大的进展，但在大量决策顺序和同时进行的情况下，对最优性和错误控制问题的理论研究却很少。这些问题是现代统计学的中心议题之一;因此，必须开发坚实的理论和强大的数据驱动方法，以帮助理解，调节和优化稀疏信号和模式恢复的动态决策过程。本提案的具体研究目标是：研究最优理论，并为信号检测、多重测试和模式分类中的各种相关问题开发数据驱动方法;为有效和准确的信号恢复开发数据采集、资源分配和决策的动态方案;并为大规模同步和顺序推理开发复合决策理论框架。 数据筛选和模式识别问题可能来自广泛的科学应用，如生物信息学、金融、信号和语言处理、图像分析以及地理和天文调查。这些问题大大促进了一个新的和活跃的跨学科的研究领域，在数据挖掘，吸引了应用数学家，统计学家和计算机科学家的极大兴趣的快速增长。该研究为这些问题中的一些基本问题提供了重要的见解，例如如何在不丢失许多信号的情况下显着减少大数据集的大小，如何最佳地将信号与噪声分离，如何准确识别不同对象的形状和模式，以及如何有效地控制大量决策中的错误膨胀。将开发方便用户的软件，免费提供给公众使用。研究人员将通过为年轻的南加州大学统计计划开发新课程，并通过指导和培训本科生和研究生，帮助他们有效地参与信息时代的教育活动。

The accurate and reliable recovery of sparse signals in massive and complex data has been a fundamental question in many scientific fields. The discovery process usually involves an extensive screening through a large number of hypotheses to separate signals of interest and also recognize their patterns. The situation can be described as finding needles of various shapes in a haystack. Despite the enormous progress on methodological work in data screening, pattern recognition and related fields, there have been little theoretical studies on the issues of optimality and error control in situations where a large number of decisions are made sequentially and simultaneously. These issues are among the central topics in modern Statistics; hence it is imperative to develop solid theory and powerful data-driven methods to help understand, regulate and optimize the dynamic decision process of sparse signal and pattern recovery. The specific research goals in this proposal are: to study the optimality theory and develop data-driven methods for a broad class of interrelated problems in signal detection, multiple testing and pattern classification; to develop a dynamic scheme for data acquisition, resource allocation and decision making for effective and accurate signal recovery; and to develop a compound decision theoretic framework for large-scale simultaneous and sequential inference. The data screening and pattern recognition problems may arise from a wide range of scientific applications such as bioinformatics, finance, signal and language processing, image analysis, and geographical and astronomical surveys. These problems have significantly contributed to the rapid growth of a new and active interdisciplinary research area in data mining that has attracted substantial interests from applied mathematicians, statisticians and computer scientists. The proposed research provides important insights on some fundamental issues in these problems such as how the size of large data sets can be reduced significantly without losing many signals, how the signals can be separated from noise optimally, how the shapes and patterns of different objects can be recognized accurately, and how the inflation of errors in a large number of decisions can be controlled effectively. User-friendly software will be developed and made freely available for public use. The investigator will integrate the proposed research into educational activities through developing new courses for the young USC Statistics program, and through mentoring and training both undergraduate and graduate students to help them participate effectively in an information era overwhelmed by massive data.