CAREER: Simultaneous and Sequential Inference of High-dimensional Data with Sparse Structure

在海量复杂数据中准确、可靠地恢复稀疏信号一直是许多科学领域的基本问题。发现过程通常包括通过大量假设进行广泛筛选，以分离感兴趣的信号并识别其模式。这种情况可以被描述为在干草堆中找到各种形状的针。尽管在数据筛选、模式识别和相关领域的方法学工作取得了巨大进展，但在顺序和同时做出大量决策的情况下，关于最优性和错误控制问题的理论研究很少。这些问题是现代统计学的中心议题之一；因此，必须发展坚实的理论和强大的数据驱动方法来帮助理解、调节和优化稀疏信号和模式恢复的动态决策过程。本课题的具体研究目标是：研究信号检测、多重测试和模式分类等一系列相互关联的问题的最优性理论和数据驱动方法；制定数据采集、资源分配和决策的动态方案，以有效和准确地恢复信号；并为大规模同时推理和顺序推理开发了一个复合决策理论框架。数据筛选和模式识别问题可能出现在广泛的科学应用中，如生物信息学、金融、信号和语言处理、图像分析、地理和天文调查。这些问题极大地促进了数据挖掘这一新的、活跃的跨学科研究领域的快速发展，吸引了应用数学家、统计学家和计算机科学家的极大兴趣。本文提出的研究为这些问题中的一些基本问题提供了重要的见解，例如如何在不丢失许多信号的情况下显著减小大数据集的大小，如何最优地将信号与噪声分离，如何准确识别不同物体的形状和模式，以及如何有效控制大量决策中的错误膨胀。将开发方便用户使用的软件，并免费提供给公众使用。研究者将通过为年轻的南加州大学统计项目开发新课程，并通过指导和培训本科生和研究生，帮助他们有效地参与被海量数据淹没的信息时代，将拟议的研究整合到教育活动中。

The accurate and reliable recovery of sparse signals in massive and complex data has been a fundamental question in many scientific fields. The discovery process usually involves an extensive screening through a large number of hypotheses to separate signals of interest and also recognize their patterns. The situation can be described as finding needles of various shapes in a haystack. Despite the enormous progress on methodological work in data screening, pattern recognition and related fields, there have been little theoretical studies on the issues of optimality and error control in situations where a large number of decisions are made sequentially and simultaneously. These issues are among the central topics in modern Statistics; hence it is imperative to develop solid theory and powerful data-driven methods to help understand, regulate and optimize the dynamic decision process of sparse signal and pattern recovery. The specific research goals in this proposal are: to study the optimality theory and develop data-driven methods for a broad class of interrelated problems in signal detection, multiple testing and pattern classification; to develop a dynamic scheme for data acquisition, resource allocation and decision making for effective and accurate signal recovery; and to develop a compound decision theoretic framework for large-scale simultaneous and sequential inference. The data screening and pattern recognition problems may arise from a wide range of scientific applications such as bioinformatics, finance, signal and language processing, image analysis, and geographical and astronomical surveys. These problems have significantly contributed to the rapid growth of a new and active interdisciplinary research area in data mining that has attracted substantial interests from applied mathematicians, statisticians and computer scientists. The proposed research provides important insights on some fundamental issues in these problems such as how the size of large data sets can be reduced significantly without losing many signals, how the signals can be separated from noise optimally, how the shapes and patterns of different objects can be recognized accurately, and how the inflation of errors in a large number of decisions can be controlled effectively. User-friendly software will be developed and made freely available for public use. The investigator will integrate the proposed research into educational activities through developing new courses for the young USC Statistics program, and through mentoring and training both undergraduate and graduate students to help them participate effectively in an information era overwhelmed by massive data.