A suite of new nonparametric methods for missing data and data from heterogeneous sources based on the theory of Frechet classes

从临床试验的传统设置到许多现代应用领域中技术驱动的大量数据收集，统计学家的原材料经常受到数据缺失的困扰。不管这是由于不响应，还是数据源的异构性日益增加，不完整性在实践中通常是不可避免的。绝大多数统计程序的设计目的是提供完整的资料，没有这些资料就可能变得不适用、无法解释或不可靠。然而，将注意力限制在完整的情况下，即没有缺失变量的数据点，通常会大大降低数据集的效用，因为在不完整的情况下会丢弃有用的信息，并且由于完整的情况没有提供总体的代表性样本而引入偏差的可能性。当从业者遇到丢失的数据时，他们必须问自己的第一个问题是数据丢失的机制，以及丢失是否会在他们的数据集分析和结果解释中造成严重问题。如果某些变量信息的缺失可以被建模为与数据值无关，那么数据就被称为完全随机缺失（MCAR），随后的分析就会比其他情况简单得多。然而，在没有适当基础的情况下做出这种假设的后果可能是严重的。我们将首先对MCAR假设的结果进行严格的研究，提出这一性质的新特征，并提供与其他领域研究的概念的新联系，包括copula理论、凸和计算几何。利用这些学科的知识，我们将为统计学家设计新的工具，为不完整数据的分析带来新的视角，并为缺失研究开辟新的领域。具体来说，我们将把MCAR的属性与fr<s:1>类和兼容性联系起来。有了必要的框架，我们将为MCAR的假设引入假设检验。在第一个实例中，这些将适用于列联表，但它们将通过分组扩展到连续数据。某些替代方案与零无法区分，但我们将证明这些测试对所有可区分的固定替代假设都有能力，并给出它们具有最优能力的情况。尽管这是至关重要的第一步，但MCAR的假设往往限制太大，无法在实践中发挥作用。然而，这种缺失可能可以用某些完全观测变量（CDM）来解释。使用从条件独立性测试问题中获得的额外见解，我们可以扩展我们早期的工作来测试这个更灵活的假设，它类似于通常的MAR假设，但比它更强大。在高维环境中，使用这种灵活的测试可能会导致低功耗，并且我们仅限于简单的测试。为了避免这个问题，我们的下一个目标将是定义和分析问题的一个宽松版本的新测试，它只试图找到从均值和协方差矩阵不相容中表现出来的null的偏离。我们将证明，即使维度在样本量中呈多项式增长，所有这些偏离都可以被检测到。一旦进行了假设检验并形成了合理的假设，从业者通常会想要进行推断，例如有信心地估计未知量。在我们提供的框架中，线性估计的置信区间的构造与我们考虑的测试问题是对偶的。我们将新技术与经验过程理论相结合，提供最小宽度置信区间，即使在不可能进行一致估计的情况下也是如此。

From the traditional settings of clinical trials to the technologically-driven mass collection of data in many modern application areas, the statistician's raw material is often plagued with missing data. Whether this be down to nonresponse, or the increasing heterogeneity of data sources, incompleteness is typically unavoidable in practice. The vast majority of statistical procedures are designed for use with complete information, and without it may become inapplicable, uninterpretable or unreliable. Restricting attention to complete cases, i.e. data points without missing variables, however, will often drastically reduce the utility of a data set, both by throwing away useful information in the non-complete cases, and by introducing the possibility of bias due to the complete cases not providing a representative sample of the population. When a practitioner encounters missing data, the first questions they must ask themselves concern the mechanism by which the data came to be missing, and whether the missingness will cause serious problems in the analysis of their data set and the interpretation of their results. If the absence of information on certain variables can be modelled as independent of the value of the data, then the data is said to be Missing Completely at Random (MCAR), and subsequent analysis is significantly simpler than it would otherwise be. However, the consequences of making this assumption without proper basis can be severe.We will begin with a rigorous study of the consequences of the MCAR assumption, presenting new characterisations of this property and providing novel connections to concepts studied in other fields, including copula theory and convex and computational geometry. Leveraging knowledge developed in these disciplines, we will design new tools for statisticians, bring new perspectives to the analysis of incomplete data, and open up new frontiers in the study of missingness. Specifically, we will link the property of MCAR to Fréchet classes and compatibility.With the necessary framework in place, we will introduce hypothesis tests for the assumption of MCAR. In the first instance these will be applicable to contingency tables, but they will be extended to continuous data through binning. Certain alternatives are indistinguishable from the null, but we will show that these tests have power against all fixed alternative hypotheses that are distinguishable, and give situations in which they have optimal power.Although a crucial first step, the assumption of MCAR is often too restrictive to be useful in practice. However, it may be that the missingness can be explained by certain fully-observed variables (CDM). Using additional insights from the problem of conditional independence testing we may extend our earlier work to test this more flexible assumption that is similar to, though stronger than, the usual MAR assumption.In high-dimensional settings, the use of such flexible tests is likely to result in low power and we are limited to simple tests. To circumvent this issue, our next goal will be to define and analyse new tests in a relaxed version of the problem, which only attempt to find departures from the null that manifest in incompatibility of means and covariance matrices. We will show that all such departures can be detected, even when dimension grows polynomially in the sample size.Once hypothesis tests have been carried out and reasonable assumptions developed, a practitioner will typically want to perform inference such as estimating an unknown quantity with confidence. In the framework we provide, the construction of confidence intervals for linear estimands is dual to the testing problems we consider. We combine our new technology with empirical process theory to provide minimal width confidence intervals, even in settings where consistent estimation is not possible.