Factor Based Imputation of Missing Data

在物理和社会科学研究中经常出现观测缺失现象；缺失的观察可能是由于人们对问题没有反应。数据定义、数据收集技术、自然灾害和战争等方面的变化。人们提出了许多方法来推测缺失的观测值；这些方法通常对缺失值的性质施加限制性假设，随机缺失是最常见的。虽然它们在实践中工作得很好，但输入数据的理论性质还没有得到很好的理解。如果没有分布理论，则无法测量输入值周围的采样不确定性。此外，假设缺失的观测是随机发生的，这通常不适用于经济数据。该项目将开发程序，使用时间和横截面维度的信息来恢复数据集中缺失的值。该框架在许多领域都有应用，但本项目使用三个领域来说明这些方法的有用性。这些方法使研究人员能够处理缺少观察的数据，从而对项目进行更有效的评估，并为企业和企业提供更好的建议。这也将有助于确立美国在大数据计量经济学领域的全球领导者地位。该项目将开发一套基于因素的估算（FBI）程序，该程序将使用时间和横截面维度的信息来恢复缺失的值。其主要思路是将数据组织成块，然后利用块之间的重叠信息。为了代替随机缺失，分析假设数据具有强因子结构。在最简单的情况下，当丢失的数据以有组织的方式出现时，解决方案是一种不超过两个主成分应用程序的算法。当缺失的数据是无序的，主成分的一个应用加上一系列的预测也将产生一致的估计。该项目将建立缺失值的收敛速率，表征采样误差，并通过模拟记录其有限样本特性。该框架具有广泛的应用，但该项目侧重于三个方面：(i)对被治疗者的个体和平均治疗效果的估计；（ii）当数据缺失影响第二矩计算时的协方差结构估计（如盈余动态）；（三）混合频率数据和在预测中使用输入值。一套用R语言编写的计算机程序将被公开。这项研究将确立美国在大数据分析领域的全球领先地位。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Missing observations occur in physical and social science research; Missing observations may arise from people not responding to question,; changes in data definitions, technology of data collection, and natural disasters and wars, among others. Many methods have been proposed to impute the missing observations; these methods often impose restrictive assumptions about the nature of the missing values, with missing at random being the most common. Though they work well in practice, the theoretical properties of the imputed data are not well understood. Without a distribution theory, the sampling uncertainty around the imputed values cannot be measured. Furthermore, the assumption that the missing observations happen at random is often not appropriate for economic data. This project will develop procedures that will use information in the time and cross-section dimensions to recover the missing values in a data set. This framework has applications in many areas but the project uses three areas to illustrate the usefulness of these methods. These methods allow researchers to work with data that has missing observations, allowing for more efficient evaluation of programs as well as provide better advise to businesses and businesses. This will also help establish the US as the global leader in big data econometrics. This project will develop a suite of factor based imputation (FBI) procedures that will use information in the time and cross-section dimensions to recover the missing values. The main insight is to organize the data into blocks and subsequently exploit the overlapping information between blocks. In place of the missing at random, the analysis assumes that the data admit a strong factor structure. In the simplest case when the missing data appear in an organized manner, the solution is an algorithm that involves no more than two applications of principal components. When the missing data are disorganized, one application of principal components coupled with a series of projections will also yield consistent estimates. The project will establish the convergence rates of the missing values, characterize the sampling error, and document its finite sample properties via simulations. This framework has broad applications but the project focuses on three: (i) estimation of the individual and average treatment effect of the treated; (ii) covariance structure estimation (of e.g. earnings dynamics) when missing data affects computation of second moments; and (iii) mixed frequency data and the use of imputed values in forecasting. A set of computer programs, written in R package, will be made publicly available. This research will establish the US as the global leader in big data analysis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.