Planes of Change: New Statistical Methods for Complex Non-Standard Systems

该项目旨在开发新的统计方法，用于分析各种领域的系统，如个性化医疗、互联网流量和经济学，这些领域会出现尖锐的阈值效应。当一个系统遭受突然冲击时，通常会经历这种剧烈影响（例如，政治紧张对股票价格的影响，社会政治动荡对社交媒体网络的影响，或医疗干预对疾病进展的影响）。这些急剧的变化对这些不同领域的从业者至关重要，因为它们通常对未来的决策有重要的影响。例如，统计学家通过所谓的“变化点”来模拟这种时间上的急剧变化；当由于多个变量同时影响而发生急剧变化时，这些区域用“变化平面”来描述。该项目旨在开发新的方法来识别问题中的这些变化点或变化面，其中大量数据现已成为可用的标准，因为存储能力和收集机制的进步，此外，记录数据的变量数量也非常大。这些方法的性能将使用数学理论和计算机生成的模拟进行仔细分析，并且这些方法也将在来自各种来源的真实数据上进行验证。预计该研究结果将对自然科学和社会科学领域产生影响。这个项目的首要主题是在一类问题中开发方法和推理，在这些问题中，阈值或边界（在一个或多个维度中）会在统计模型或估计范式中自然产生不连续。这些问题既可以在大量数据的情况下进行研究，也可以在协变量数量超过观测值数量的情况下进行研究。一维中考虑的边界是变化点，而多维中考虑的边界是超平面。所研究的问题呈现出两种不同类型的复杂性：(a)大量可用数据，和/或(b)相对于观测数量的大量协变量。特别是：(i)开发了许多想法，用于从（回顾性观察）长时间序列中进行智能采样，以通过只需要分析整个序列的消失部分（从而提供计算效益）的程序确定多个变化点的位置，但产生的估计在精度上与分析整个序列所获得的标准估计相匹配。这个想法被扩展到多维的基于协变量的回归/似然模型，其中回归或似然的参数在协变量空间的超平面的两侧是不同的。（ii）研究了模型结构或待优化准则函数中涉及高维协变量的超平面问题，并研究了新的变量选择和估计方法。从应用的角度来看，这里考虑的问题很重要，但很难，因为高维范式必须扩展到本质上不连续的设置，超出（几乎）平方根n率。对这些问题的有效解决将推进这些重要系统类别的统计方法。

The project aims to develop new statistical methodologies for analysis of systems in a variety of fields, such as personalized medicine, internet traffic, and economics, in which sharp threshold effects occur. Such sharp effects are typically experienced when a system is subjected to a sudden shock (e.g., the effect of political tension on stock prices, the effect of socio-political upheaval on social-media networks, or the effect of a medical intervention on disease progression). Such sharp changes are of critical interest to practitioners in these different fields as they typically have important implications for future decision-making. Statisticians model such sharp changes in time, for example, through what are called "change-points;" when the sharp change happens due to the effect of multiple variables simultaneously, such regions are described in terms of "change-planes." This project aims to develop novel methods of identifying such change-points or change-planes in problems where massive amounts of data -- which have now become the norm given advances in storage capabilities as well as collection mechanisms -- are available, and furthermore, the number of variables on which data are recorded is also very large. The performance of such methods will be carefully analyzed using mathematical theory as well as computer-generated simulations, and the methods will also be validated on real data coming from a variety of sources. It is anticipated that the results of the research will have impact in a variety of natural science as well as social science disciplines. The overarching theme of this project is to develop methodology and inference in a class of problems in which thresholds or boundaries (in one or multiple dimensions) that induce discontinuities arise naturally, either in the statistical model or in the estimation paradigm. The problems are studied both in the setting of massive amounts of data as well as in scenarios where the number of covariates can exceed the number of observations. The boundaries considered in one-dimension are change-points, while those in multiple dimensions are hyper-planes. The studied problems present two different kinds of complexities: (a) massive amounts of available data, and/or (b) large numbers of covariates relative to number of observations. In particular: (i) A number of ideas are developed for sampling intelligently from (retrospectively observed) long time-series to determine the locations of multiple change-points via procedures that require analyzing only a vanishing fraction of the entire series (thereby providing computational benefits), yet produce estimates that match, in precision, the standard estimates that would have been obtained analyzing the entire series. This idea is extended to regression/likelihood based models with covariates in multiple dimensions where the parameters of the regression or the likelihood are different on either side of a hyper-plane in covariate space. (ii) Problems involving hyper-planes, either in the structure of the model or in the criterion function to be optimized, with high-dimensional covariates are studied and new variable selection and estimation methods are investigated. The problems under consideration here are important from the perspective of applications but difficult because the high-dimensional paradigm has to be extended to intrinsically discontinuous settings, outside the (almost) square-root-n rate. Effective solutions to these problems will advance statistical methodology for these important classes of systems.