Semiparametric Estimation and Variable Selection in the Presence of Nonignorable Nonresponse

在许多统计应用中都存在无响应现象。在大多数调查问题中，许多抽样单位无法提供部分或全部调查问题的答案。在医学或健康研究中，不完整数据的百分比通常是可观的。当无响应与丢失的数据相关时，处理无响应是非常具有挑战性的。由于这项研究的动机是美国人口普查局和加拿大统计局等调查机构的问题，或者是医学和健康研究中的数据集，因此本研究的结果将对实践中处理估计和推断的无应答的方法产生重大影响。这项研究的结果也将为这一领域的进一步研究提供参考。当无反应机制或倾向仅取决于观察数据时，无反应称为可忽略；否则，它是不可忽略的。关于处理可以忽略的无反应的方法论，已经有了丰富的文献。处理不可忽略的无响应更具挑战性，因为必须施加假设以确保未知总体特征的可辨识性和可估计性，并且这些假设很难使用具有不可忽略的无响应的数据来检验。对于具有不可忽略无响应的数据，应用为可忽略无响应而开发的方法可能会在统计估计和推断中产生严重的偏差。在以下两个一般主题中，本研究侧重于基于不可忽略无响应的数据的估计。(1)半参数估计。如果基于无响应倾向和感兴趣的总体分布假设一个完全参数模型，则在某些可辨识性假设下，可以使用参数似然得到感兴趣参数的有效估计器。然而，这种参数方法对模型的错误指定很敏感，特别是当无响应是不可忽略的时候。另一方面，与没有无反应的情况不同，纯粹的非参数方法不能识别总体。本研究研究半参数方法，假设倾向或总体分布的一个分量是参数的，其他的是非参数的。在Meta分析中，将努力研究不同假设下各种方法的稳健性和有效性，包括不可忽视的无反应的纵向或多变量结果，具有缺失结果和协变量的问题，以及未测量的混杂因素或系统性缺失的协变量数据。(2)模型和变量的选择。当无响应不可忽略时，需要使用称为无响应工具的协变量，它总是被观察并有助于识别总体参数。此外，还必须假定倾向或人口分布的参数分量。因此，希望执行模型和/或变量选择，以确保假定的参数分量和所选择的无响应仪器是适当的。由于不可忽略的无响应，现有的模型和变量选择技术不再适用。这项研究将为模型选择和无响应仪器的选择开发新的技术。此外，在大数据时代，存在着极其庞大的可用作协变量的辅助变量集，本研究将研究在存在不可忽视的无响应的情况下，为准确估计而进行的降维和变量选择。

Nonresponse exists in many statistical applications. In most survey problems, many sampled units fail to provide answers to some or all survey questions. In medical or health studies, the percentages of incomplete data are often appreciable. Handling nonresponse is very challenging when nonresponse is related to the missing data. Since this research is motivated by problems in survey agencies such as the U.S. Census Bureau and Statistics Canada, or by data sets in medical and health studies, results obtained from this research will have significant impacts on the methodology of handling nonresponse for estimation and inference in practice. The results from this research will also shed light on further research in this area. When the nonresponse mechanism or propensity depends on observed data only, the nonresponse is called ignorable; otherwise, it is nonignorable. There is a rich literature on methodology of handling ignorable nonresponse. Handling nonignorable nonresponse is much more challenging, since assumptions have to be imposed to ensure the identifiability and estimability of unknown population characteristics and these assumptions are hard to check using data with nonignorable nonresponse. Applying methods developed for ignorable nonresponse to data with nonignorable nonresponse may create serious biases in statistical estimation and inference. This research focuses on estimation based on data with nonignorable nonresponse in the following two general topics. (1) Semiparametric estimation. If a fully parametric model is assumed on the nonresponse propensity and the population distribution of interest, then valid estimators of parameters of interest may be derived using the parametric likelihood under some identifiability assumption. However, this parametric approach is sensitive to model misspecification, especially when nonresponse is nonignorable. On the other hand, unlike the situation with no nonresponse, a purely nonparametric approach cannot identify the population. This research studies semiparametric methods, assuming one component of the propensity or population distribution is parametric and the others are nonparametric. Efforts will be made to study robustness and efficiency of various methods under different assumptions, longitudinal or multivariate outcomes with nonignorable nonresponse, problems with both missing outcomes and covariates, and unmeasured confounders or systematic missing covariate data in meta analyses. (2) Model and variable selection. When nonresponse is nonignorable, a covariate called nonresponse instrument needs to be used, which is always observed and helps to identify population parameters. In addition, a parametric component of either the propensity or the population distribution has to be assumed. Thus, it is desired to perform model and/or variable selection to ensure that the assumed parametric component and the selected nonresponse instrument are appropriate. Because of nonignorable nonresponse, the existing model and variable selection techniques are not applicable. This research will develop new techniques for model selection and the selection of nonresponse instruments. Furthermore, in the big data era there exists an extremely large set of auxiliary variables that can be used as covariates and this research will study dimension reduction and variable selection for accurate estimation in the presence of nonignorable nonresponse.