III: Small: Causal and Statistical Inference in the Presence of Confounding Factors

技术：如果混杂因素与观察到的数据相关，则未测量的混杂因素的存在可能会导致错误的统计和因果推断。这种现象至少在两个重要的应用中得到了充分的记录。一项应用是从相关个体群体中识别与疾病有关的遗传变异。第二个应用是在比较疾病和健康样本时识别疾病中活跃的基因。在本提案中，我们提出了一种新方法，利用对混杂因素如何影响高维数据的洞察来纠正未观察到的混杂因素。这些见解激发了对特定类型的混杂因素的正式定义，我们将其称为“低级混杂因素”。形式化这个定义使我们能够激发纠正这些类型混杂因素影响的方法，即使没有观察到混杂因素。我们的提案将发展一种关于混杂因素如何影响数据以及在什么条件下可以纠正未观察到的混杂因素的理论。所提出的理论与理解稀疏性的最新进展有关，稀疏性在电气工程、计算机科学和统计学中已得到充分研究。我们提出的方法的结果将为存在此类混杂因素的应用带来改进的方法。 非技术：从高维数据推断知识是影响几乎所有科学领域的基本问题，包括物理、天文学、化学、计算机科学、社会科学和生物学的许多领域。其中许多问题是由最近可用的大量数据源以及测量或数据收集技术的进步造成的。一个主要的挑战是存在未知（且无法测量）的混杂因素。混杂因素是通常在数据中观察不到的变量，但与数据的各种特征相关。不幸的是，混杂因素可能会导致错误的推论。这种现象已在至少两个重要应用中得到充分证明：一个应用是从相关个体群体中识别与疾病有关的遗传变异，第二个应用是在比较疾病和健康样本时识别疾病中活跃的基因。如果在数据中观察到混杂因素，可以使用传统方法进行推理。然而，处理未观察到的混杂因素更加困难。该项目将开发和研究一种新方法来纠正未观察到的混杂因素，利用对混杂因素如何影响高维数据的见解。该项目通过向本科生和研究生提供跨学科研究机会以及软件和数据的分发，在广泛的科学问题上发挥了实用作用，因此产生了广泛的影响。

Technical:The presence of unmeasured confounding factors can result in incorrect statistical and causal inferences if the confounding factors are correlated with the observed data. This phenomenon has been well documented in at least two important applications. One application is identifying genetic variation involved in disease from populations of related individuals. A second application is identifying genes active in a disease when comparing disease and health samples. In this proposal we propose a new approach to correct for unobserved confounders in taking advantage of insights into how confounders affect high dimensional data. These insights motivate a formal definition for a specific type of confounder which we term a 'low-rank confounder.' Formalizing this definition allows us to motivate methods for correcting for the effects of these types confounders even when the confounders are not observed. Our proposal will develop a theory of how confounders affect data and under what conditions unobserved confounders can be corrected. The proposed theory is related to recent developments in understanding sparsity which has been well studied in electrical engineering, computer science and statistics. The result of our proposed methods will lead to improved methods for applications where such confounders are present.Non-technical:Inference of knowledge from high dimensional data is a fundamental problem affecting virtually all areas of science including physics, astronomy, chemistry, computer science, social science and many areas of biology. Many of these problems are driven by recently available large sources of data and advances in measurement or data collection technologies. A major challenge is the presence of unknown (and unmeasured) confounding factors. Confounding factors are variables that are often not observed in the data, but are correlated with various features of the data. Unfortunately, confounding factors can cause incorrect inferences. This phenomenon has been well documented in at least two important applications: one application is identifying genetic variation involved in disease from populations of related individuals, and a second application is identifying genes active in a disease when comparing disease and health samples. There are traditional approaches to perform inference if the confounders are observed in the data. However, dealing with unobserved confounders is more difficult. This project will develop and study a new approach to correct for unobserved confounders, taking advantage of insights into how confounders affect high dimensional data. The project has broad impact due to its utility in a wide range of scientific questions, through the interdisciplinary research opportunities provided to undergraduate and graduate students, and through the distribution of software and data.