Planes of Change: New Statistical Methods for Complex Non-Standard Systems

该项目旨在开发新的统计方法，用于分析各种领域的系统，如个性化医疗、互联网流量和经济学，在这些领域中，会出现明显的门槛效应。当一个系统受到突然冲击(例如，政治紧张局势对股价的影响、社会政治动荡对社交媒体网络的影响，或医疗干预对疾病进展的影响)时，通常会经历这种尖锐的影响。这种急剧的变化对这些不同领域的从业者来说是至关重要的，因为它们通常对未来的决策具有重要影响。统计学家在时间上模拟这种急剧的变化，例如，通过所谓的“变化点”；当急剧的变化由于多个变量的影响同时发生时，这些区域被用“变化面”来描述。这个项目旨在开发新的方法，在大量数据--由于存储能力和收集机制的进步--现在已经成为常态，而且记录数据的变量也非常多的情况下，确定这样的变化点或变化面。这些方法的性能将使用数学理论和计算机生成的模拟进行仔细分析，并将在来自各种来源的真实数据上进行验证。预计研究结果将对各种自然科学和社会科学学科产生影响。该项目的主要主题是在一类问题中开发方法和推理，在这类问题中，导致不连续的阈值或边界(在一个或多个方面)自然地出现，无论是在统计模型中还是在估计范例中。在设置大量数据的情况下以及在协变量数量可能超过观测数量的情况下都研究了这些问题。一维边界是变点，多维边界是超平面。所研究的问题呈现出两种不同的复杂性：(A)大量的可用数据，和/或(B)相对于观测数量的大量协变量。特别是：(1)提出了从(回溯观察到的)长时间序列进行智能抽样的一些想法，以便通过只需要分析整个序列的一小部分消失部分的程序来确定多个变化点的位置(从而提供计算效益)，同时产生在精度上与分析整个序列所获得的标准估计相匹配的估计。将这一思想推广到协变量为多维的基于回归/似然的模型，其中回归的参数或似然在协变量空间中的超平面的两侧是不同的。(2)研究了具有高维协变量的超平面问题，无论是在模型结构中还是在待优化的准则函数中，都研究了新的变量选择和估计方法。从应用的角度来看，这里考虑的问题很重要，但很困难，因为高维范例必须扩展到本质上不连续的设置，超出(几乎)平方根-n比率。对这些问题的有效解决将推动这些重要类别系统的统计方法。

The project aims to develop new statistical methodologies for analysis of systems in a variety of fields, such as personalized medicine, internet traffic, and economics, in which sharp threshold effects occur. Such sharp effects are typically experienced when a system is subjected to a sudden shock (e.g., the effect of political tension on stock prices, the effect of socio-political upheaval on social-media networks, or the effect of a medical intervention on disease progression). Such sharp changes are of critical interest to practitioners in these different fields as they typically have important implications for future decision-making. Statisticians model such sharp changes in time, for example, through what are called "change-points;" when the sharp change happens due to the effect of multiple variables simultaneously, such regions are described in terms of "change-planes." This project aims to develop novel methods of identifying such change-points or change-planes in problems where massive amounts of data -- which have now become the norm given advances in storage capabilities as well as collection mechanisms -- are available, and furthermore, the number of variables on which data are recorded is also very large. The performance of such methods will be carefully analyzed using mathematical theory as well as computer-generated simulations, and the methods will also be validated on real data coming from a variety of sources. It is anticipated that the results of the research will have impact in a variety of natural science as well as social science disciplines. The overarching theme of this project is to develop methodology and inference in a class of problems in which thresholds or boundaries (in one or multiple dimensions) that induce discontinuities arise naturally, either in the statistical model or in the estimation paradigm. The problems are studied both in the setting of massive amounts of data as well as in scenarios where the number of covariates can exceed the number of observations. The boundaries considered in one-dimension are change-points, while those in multiple dimensions are hyper-planes. The studied problems present two different kinds of complexities: (a) massive amounts of available data, and/or (b) large numbers of covariates relative to number of observations. In particular: (i) A number of ideas are developed for sampling intelligently from (retrospectively observed) long time-series to determine the locations of multiple change-points via procedures that require analyzing only a vanishing fraction of the entire series (thereby providing computational benefits), yet produce estimates that match, in precision, the standard estimates that would have been obtained analyzing the entire series. This idea is extended to regression/likelihood based models with covariates in multiple dimensions where the parameters of the regression or the likelihood are different on either side of a hyper-plane in covariate space. (ii) Problems involving hyper-planes, either in the structure of the model or in the criterion function to be optimized, with high-dimensional covariates are studied and new variable selection and estimation methods are investigated. The problems under consideration here are important from the perspective of applications but difficult because the high-dimensional paradigm has to be extended to intrinsically discontinuous settings, outside the (almost) square-root-n rate. Effective solutions to these problems will advance statistical methodology for these important classes of systems.