EAGER: Deep Causal Representation Learning for Generalizable Visual Understanding

深度学习的最新发展导致了计算机视觉和其他领域的显着进步。 然而，目前的计算机视觉系统在与其训练环境不同的新环境中表现不佳。 如果不解决这个问题，可能会导致预测错误，从而对自动驾驶、机器人和制造等生命攸关和高风险应用造成重大后果。 为了应对这一挑战，该项目引入了一种使用潜在特征的新型深度因果学习框架。 与大多数可观察的特征不同，潜在特征是隐藏的。例如，在对象识别任务中，潜在因子可以是描述从图像的像素和每个训练图像的标签中提取的对象的比例和姿态的向量。该框架利用深度学习来学习潜在特征表示，并利用因果学习来执行特征表示，以形成跨域不变的目标变量的直接原因和影响。 除了使计算机视觉系统能够安全地部署在开放世界环境中之外，该项目还可以显着推进人工智能（AI）和机器学习的其他领域，包括迁移学习，领域适应和终身学习。该项目将研究与教育相结合，并通过利用伦斯勒理工学院的现有项目，让本科生参与这项研究。这项研究探索了将因果关系融入深度神经网络以增强泛化，公平和解释的新想法。该项目采用因果马尔可夫毯（CMB）捕捉数据中的潜在因果机制。 它进一步证明了CMB特征在预测目标变量时的最佳特性。 最后，它引入了一个端到端的深度学习框架，可以在没有因果充分性假设的情况下有效地学习目标变量的CMB特征。 该项目将评估一些计算机视觉任务的框架。尽管有巨大的潜力，但该项目研究了计算机视觉和机器学习中相对较少探索的研究领域。 该项目的目标是产生初步的结果，以证明其可行性，并确定改进的领域。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

The recent developments in deep learning have led to remarkable progress in computer vison and other fields. Current computer vision systems, however, do not perform well in a new environment different from its training environment. This problem, if left unaddressed, could lead to predictive errors and hence significant consequences for life-critical and high-stakes applications such as autonomous driving, robotics, and manufacturing. To address this challenge, this project introduces a novel deep causal learning framework using latent features. Unlike most features that are observable, latent features are hidden. For example, in an object recognition task, latent factors can be vectors describing the scale and poses of objects extracted from the image’s pixels and labels of every training image. The framework leverages deep learning to learn latent feature representation and causal learning to enforce the feature representation to form the direct causes and effects of the target variable that are invariant across domains. In addition to enable computer vision systems to be safely deployable in open-world environments, this project can significantly advance other fields of artificial intelligence (AI) and machine learning, including transfer learning, domain adaptation, and lifelong learning. The project will integrate research with education and involve undergraduate students in this research by leveraging the existing programs at Rensselaer Polytechnic Institute.This research explores a novel idea of integrating causality into deep neural networks to enhance generalization, fairness, and explanation. The project employs the Causal Markov Blanket (CMB) to capture the underlying causal mechanism in the data. It further demonstrates the optimal properties of the CMB features in predicting the target variable. Finally, it introduces an end-to-end deep learning framework to efficiently learn the CMB features for a target variable without causal sufficiency assumption. The project will evaluate the framework on some computer vision tasks. Despite its significant potential, this project investigates a relatively less explored area of study in computer vision and machine learning. The goal of the project is to generate preliminary results to demonstrate its feasibility and to identify areas of improvements.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.