A suite of new nonparametric methods for missing data and data from heterogeneous sources based on the theory of Frechet classes

从传统的临床试验设置，到许多现代应用领域由技术驱动的海量数据收集，统计学家的原材料经常被缺失的数据所困扰。无论这是由于无响应，还是由于数据源的日益异构性，不完整性在实践中通常是不可避免的。绝大多数统计程序都是为使用完整信息而设计的，如果没有完整的信息，可能会变得不适用、无法解释或不可靠。然而，将注意力局限于完整案例，即没有遗漏变量的数据点，往往会极大地减少数据集的效用，因为在非完整案例中丢弃了有用的信息，而且由于完整案例不能提供具有代表性的总体样本而可能产生偏见。当从业者遇到丢失的数据时，他们必须问自己的第一个问题是关于数据丢失的机制，以及这种丢失是否会在分析他们的数据集和解释他们的结果时造成严重的问题。如果关于某些变量的信息的缺失可以建模为独立于数据的值，那么数据在随机(MCAR)中被称为完全缺失，并且随后的分析比否则要简单得多。然而，在没有适当基础的情况下做出这一假设的后果可能是严重的。我们将从严格研究MCAR假设的后果开始，给出这一性质的新特征，并提供与其他领域研究的概念的新联系，包括Copula理论以及凸几何和计算几何。利用这些学科发展的知识，我们将为统计学家设计新的工具，为不完整数据的分析带来新的视角，并在遗漏研究中开辟新的前沿。具体地说，我们将把MCAR的属性与Fréchet类和兼容性联系起来。有了必要的框架，我们将为MCAR的假设引入假设检验。在第一种情况下，这些将适用于列联表，但它们将通过入库扩展到连续数据。某些替代方案与零是无法区分的，但我们将证明这些测试相对于所有可区分的固定替代假设具有能力，并给出它们具有最优能力的情况。尽管这是关键的第一步，但MCAR的假设往往过于严格，无法在实践中使用。然而，这种错配可能可以用某些完全观测变量(CDM)来解释。使用来自条件独立性测试问题的其他见解，我们可以扩展我们早期的工作，以测试这个更灵活的假设，它类似于通常的MAR假设，但比通常的MAR假设更强。在高维设置中，使用这种灵活的测试可能会导致低功耗，我们仅限于简单的测试。为了绕过这个问题，我们的下一个目标将是在这个问题的宽松版本中定义和分析新的测试，该测试只试图找到表现为均值和协方差矩阵不相容的与零的背离。我们将证明，即使当维度在样本量中以多项式增长时，所有这些偏离都可以被检测到。一旦进行了假设检验并建立了合理的假设，从业者通常会想要执行推理，例如自信地估计未知量。在我们提供的框架下，线性估计量的置信度区间的构造对我们所考虑的测试问题是对偶的。我们将我们的新技术与经验过程理论相结合，以提供最小宽度可信区间，即使在无法进行一致估计的情况下也是如此。

From the traditional settings of clinical trials to the technologically-driven mass collection of data in many modern application areas, the statistician's raw material is often plagued with missing data. Whether this be down to nonresponse, or the increasing heterogeneity of data sources, incompleteness is typically unavoidable in practice. The vast majority of statistical procedures are designed for use with complete information, and without it may become inapplicable, uninterpretable or unreliable. Restricting attention to complete cases, i.e. data points without missing variables, however, will often drastically reduce the utility of a data set, both by throwing away useful information in the non-complete cases, and by introducing the possibility of bias due to the complete cases not providing a representative sample of the population. When a practitioner encounters missing data, the first questions they must ask themselves concern the mechanism by which the data came to be missing, and whether the missingness will cause serious problems in the analysis of their data set and the interpretation of their results. If the absence of information on certain variables can be modelled as independent of the value of the data, then the data is said to be Missing Completely at Random (MCAR), and subsequent analysis is significantly simpler than it would otherwise be. However, the consequences of making this assumption without proper basis can be severe.We will begin with a rigorous study of the consequences of the MCAR assumption, presenting new characterisations of this property and providing novel connections to concepts studied in other fields, including copula theory and convex and computational geometry. Leveraging knowledge developed in these disciplines, we will design new tools for statisticians, bring new perspectives to the analysis of incomplete data, and open up new frontiers in the study of missingness. Specifically, we will link the property of MCAR to Fréchet classes and compatibility.With the necessary framework in place, we will introduce hypothesis tests for the assumption of MCAR. In the first instance these will be applicable to contingency tables, but they will be extended to continuous data through binning. Certain alternatives are indistinguishable from the null, but we will show that these tests have power against all fixed alternative hypotheses that are distinguishable, and give situations in which they have optimal power.Although a crucial first step, the assumption of MCAR is often too restrictive to be useful in practice. However, it may be that the missingness can be explained by certain fully-observed variables (CDM). Using additional insights from the problem of conditional independence testing we may extend our earlier work to test this more flexible assumption that is similar to, though stronger than, the usual MAR assumption.In high-dimensional settings, the use of such flexible tests is likely to result in low power and we are limited to simple tests. To circumvent this issue, our next goal will be to define and analyse new tests in a relaxed version of the problem, which only attempt to find departures from the null that manifest in incompatibility of means and covariance matrices. We will show that all such departures can be detected, even when dimension grows polynomially in the sample size.Once hypothesis tests have been carried out and reasonable assumptions developed, a practitioner will typically want to perform inference such as estimating an unknown quantity with confidence. In the framework we provide, the construction of confidence intervals for linear estimands is dual to the testing problems we consider. We combine our new technology with empirical process theory to provide minimal width confidence intervals, even in settings where consistent estimation is not possible.