III: Small: Causal and Statistical Inference in the Presence of Confounding Factors

技术性：如果混杂因素与观测数据相关，则未测量混杂因素的存在可能导致不正确的统计和因果推断。这种现象至少在两个重要的应用中得到了充分的证明。一种应用是从相关个体群体中识别与疾病有关的遗传变异。第二个应用是在比较疾病和健康样本时识别疾病中活跃的基因。在本提案中，我们提出了一种新的方法来纠正未观察到的混杂因素，利用对混杂因素如何影响高维数据的见解。这些见解激发了对特定类型的混杂因素的正式定义，我们称之为“低秩混杂因素”。将这一定义形式化，使我们能够激发纠正这些类型混杂因素影响的方法，即使这些混杂因素没有被观察到。我们的建议将发展一个关于混杂因素如何影响数据以及在什么条件下可以纠正未观察到的混杂因素的理论。提出的理论与最近在理解稀疏性方面的发展有关，稀疏性在电气工程、计算机科学和统计学中得到了很好的研究。我们提出的方法的结果将导致改进的方法，其中这些混杂存在的应用。非技术：从高维数据中推断知识是一个影响几乎所有科学领域的基本问题，包括物理学、天文学、化学、计算机科学、社会科学和生物学的许多领域。许多这些问题是由最近可获得的大量数据来源和测量或数据收集技术的进步造成的。一个主要的挑战是存在未知的（和无法测量的）混杂因素。混杂因素是在数据中通常没有观察到的变量，但与数据的各种特征相关。不幸的是，混杂因素可能导致不正确的推断。这一现象至少在两个重要的应用中得到了充分的证明：一个应用是从相关个体群体中识别与疾病有关的遗传变异，第二个应用是在比较疾病和健康样本时识别疾病中活跃的基因。如果在数据中观察到混杂因素，则有传统的方法来执行推理。然而，处理未观察到的混杂因素更加困难。该项目将开发和研究一种新的方法来纠正未观察到的混杂因素，利用对混杂因素如何影响高维数据的见解。通过为本科生和研究生提供跨学科研究机会，以及通过软件和数据的分发，该项目具有广泛的影响，因为它在广泛的科学问题中具有实用性。

Technical:The presence of unmeasured confounding factors can result in incorrect statistical and causal inferences if the confounding factors are correlated with the observed data. This phenomenon has been well documented in at least two important applications. One application is identifying genetic variation involved in disease from populations of related individuals. A second application is identifying genes active in a disease when comparing disease and health samples. In this proposal we propose a new approach to correct for unobserved confounders in taking advantage of insights into how confounders affect high dimensional data. These insights motivate a formal definition for a specific type of confounder which we term a 'low-rank confounder.' Formalizing this definition allows us to motivate methods for correcting for the effects of these types confounders even when the confounders are not observed. Our proposal will develop a theory of how confounders affect data and under what conditions unobserved confounders can be corrected. The proposed theory is related to recent developments in understanding sparsity which has been well studied in electrical engineering, computer science and statistics. The result of our proposed methods will lead to improved methods for applications where such confounders are present.Non-technical:Inference of knowledge from high dimensional data is a fundamental problem affecting virtually all areas of science including physics, astronomy, chemistry, computer science, social science and many areas of biology. Many of these problems are driven by recently available large sources of data and advances in measurement or data collection technologies. A major challenge is the presence of unknown (and unmeasured) confounding factors. Confounding factors are variables that are often not observed in the data, but are correlated with various features of the data. Unfortunately, confounding factors can cause incorrect inferences. This phenomenon has been well documented in at least two important applications: one application is identifying genetic variation involved in disease from populations of related individuals, and a second application is identifying genes active in a disease when comparing disease and health samples. There are traditional approaches to perform inference if the confounders are observed in the data. However, dealing with unobserved confounders is more difficult. This project will develop and study a new approach to correct for unobserved confounders, taking advantage of insights into how confounders affect high dimensional data. The project has broad impact due to its utility in a wide range of scientific questions, through the interdisciplinary research opportunities provided to undergraduate and graduate students, and through the distribution of software and data.