Semiparametric Estimation and Variable Selection in the Presence of Nonignorable Nonresponse

无响应存在于许多统计应用中。在大多数调查问题中，许多抽样单位不能提供部分或全部调查问题的答案。在医学或健康研究中，不完整数据的百分比往往是可观的。当非响应与丢失的数据相关时，处理非响应是非常有挑战性的。由于本研究的动机来自于美国人口普查局和加拿大统计局等调查机构的问题，或者来自于医学和健康研究中的数据集，因此本研究的结果将对实践中处理非响应估计和推理的方法产生重大影响。这项研究的结果也将为该领域的进一步研究提供启示。当无响应机制或倾向仅依赖于观测数据时，无响应称为可忽略的；否则，它是不可忽略的。关于处理可忽略非反应的方法，已有丰富的文献。处理不可忽略的非响应更具有挑战性，因为必须施加假设以确保未知种群特征的可识别性和可估计性，而这些假设很难使用具有不可忽略的非响应的数据来检查。对具有不可忽略非响应的数据应用可忽略非响应的方法可能会在统计估计和推断中产生严重的偏差。本文的研究主要集中在以下两个一般主题中基于不可忽略无响应数据的估计。(1)半参数估计。如果在无响应倾向和兴趣的总体分布上假设了一个全参数模型，那么在某些可辨识假设下，利用参数似然可以得到兴趣参数的有效估计量。然而，这种参数化方法对模型的错误规范很敏感，特别是当非响应不可忽略时。另一方面，与没有无响应的情况不同，纯粹的非参数方法不能识别总体。本研究研究半参数方法，假设倾向或总体分布的一个组成部分是参数的，其他部分是非参数的。将努力研究各种方法在不同假设下的稳健性和效率，不可忽视的无反应的纵向或多变量结果，缺失结果和协变量的问题，以及meta分析中未测量的混杂因素或系统缺失协变量数据。(2)模型和变量选择。当非响应不可忽略时，需要使用称为非响应工具的协变量，它总是被观察到并有助于识别总体参数。此外，必须假设倾向或总体分布的参数成分。因此，需要进行模型和/或变量选择，以确保假设的参数组件和选择的无响应仪器是适当的。由于不可忽略的非响应，现有的模型和变量选择技术都不适用。本研究将为模型选择和非响应工具的选择开发新的技术。此外，在大数据时代，存在着非常大的辅助变量集，这些辅助变量集可以作为协变量，本研究将研究在不可忽略的非响应存在的情况下，为了准确估计而进行降维和变量选择。

Nonresponse exists in many statistical applications. In most survey problems, many sampled units fail to provide answers to some or all survey questions. In medical or health studies, the percentages of incomplete data are often appreciable. Handling nonresponse is very challenging when nonresponse is related to the missing data. Since this research is motivated by problems in survey agencies such as the U.S. Census Bureau and Statistics Canada, or by data sets in medical and health studies, results obtained from this research will have significant impacts on the methodology of handling nonresponse for estimation and inference in practice. The results from this research will also shed light on further research in this area. When the nonresponse mechanism or propensity depends on observed data only, the nonresponse is called ignorable; otherwise, it is nonignorable. There is a rich literature on methodology of handling ignorable nonresponse. Handling nonignorable nonresponse is much more challenging, since assumptions have to be imposed to ensure the identifiability and estimability of unknown population characteristics and these assumptions are hard to check using data with nonignorable nonresponse. Applying methods developed for ignorable nonresponse to data with nonignorable nonresponse may create serious biases in statistical estimation and inference. This research focuses on estimation based on data with nonignorable nonresponse in the following two general topics. (1) Semiparametric estimation. If a fully parametric model is assumed on the nonresponse propensity and the population distribution of interest, then valid estimators of parameters of interest may be derived using the parametric likelihood under some identifiability assumption. However, this parametric approach is sensitive to model misspecification, especially when nonresponse is nonignorable. On the other hand, unlike the situation with no nonresponse, a purely nonparametric approach cannot identify the population. This research studies semiparametric methods, assuming one component of the propensity or population distribution is parametric and the others are nonparametric. Efforts will be made to study robustness and efficiency of various methods under different assumptions, longitudinal or multivariate outcomes with nonignorable nonresponse, problems with both missing outcomes and covariates, and unmeasured confounders or systematic missing covariate data in meta analyses. (2) Model and variable selection. When nonresponse is nonignorable, a covariate called nonresponse instrument needs to be used, which is always observed and helps to identify population parameters. In addition, a parametric component of either the propensity or the population distribution has to be assumed. Thus, it is desired to perform model and/or variable selection to ensure that the assumed parametric component and the selected nonresponse instrument are appropriate. Because of nonignorable nonresponse, the existing model and variable selection techniques are not applicable. This research will develop new techniques for model selection and the selection of nonresponse instruments. Furthermore, in the big data era there exists an extremely large set of auxiliary variables that can be used as covariates and this research will study dimension reduction and variable selection for accurate estimation in the presence of nonignorable nonresponse.