EAGER: Deep Causal Representation Learning for Generalizable Visual Understanding

最近深度学习的发展在计算机视觉和其他领域取得了显著的进步。然而，当前的计算机视觉系统在与训练环境不同的新环境中表现不佳。这个问题如果不加以解决，可能会导致预测错误，从而对自动驾驶、机器人和制造业等攸关生命和高风险的应用产生重大影响。为了应对这一挑战，本项目引入了一种使用潜在特征的新型深度因果学习框架。与大多数可观察到的特征不同，潜在特征是隐藏的。例如，在对象识别任务中，潜在因素可以是描述从图像像素和每个训练图像的标签中提取的对象的规模和姿态的向量。该框架利用深度学习来学习潜在特征表示，并利用因果学习来强化特征表示，以形成目标变量的直接因果关系，这些直接因果关系是跨域不变的。除了使计算机视觉系统能够在开放世界环境中安全部署外，该项目还可以显著推进人工智能（AI）和机器学习的其他领域，包括迁移学习、领域适应和终身学习。该项目将研究与教育相结合，并利用伦斯勒理工学院现有的课程，让本科生参与这项研究。本研究探索了一种将因果关系整合到深度神经网络中的新思路，以增强泛化、公平性和解释性。该项目采用因果马尔可夫毯子（CMB）来捕捉数据中潜在的因果机制。进一步证明了CMB特征在预测目标变量方面的最佳性能。最后，引入了一个端到端深度学习框架，在没有因果充分假设的情况下，有效地学习目标变量的CMB特征。该项目将在一些计算机视觉任务上评估该框架。尽管具有巨大的潜力，但该项目调查了计算机视觉和机器学习中一个相对较少探索的研究领域。该项目的目标是产生初步结果，以证明其可行性并确定需要改进的领域。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The recent developments in deep learning have led to remarkable progress in computer vison and other fields. Current computer vision systems, however, do not perform well in a new environment different from its training environment. This problem, if left unaddressed, could lead to predictive errors and hence significant consequences for life-critical and high-stakes applications such as autonomous driving, robotics, and manufacturing. To address this challenge, this project introduces a novel deep causal learning framework using latent features. Unlike most features that are observable, latent features are hidden. For example, in an object recognition task, latent factors can be vectors describing the scale and poses of objects extracted from the image’s pixels and labels of every training image. The framework leverages deep learning to learn latent feature representation and causal learning to enforce the feature representation to form the direct causes and effects of the target variable that are invariant across domains. In addition to enable computer vision systems to be safely deployable in open-world environments, this project can significantly advance other fields of artificial intelligence (AI) and machine learning, including transfer learning, domain adaptation, and lifelong learning. The project will integrate research with education and involve undergraduate students in this research by leveraging the existing programs at Rensselaer Polytechnic Institute.This research explores a novel idea of integrating causality into deep neural networks to enhance generalization, fairness, and explanation. The project employs the Causal Markov Blanket (CMB) to capture the underlying causal mechanism in the data. It further demonstrates the optimal properties of the CMB features in predicting the target variable. Finally, it introduces an end-to-end deep learning framework to efficiently learn the CMB features for a target variable without causal sufficiency assumption. The project will evaluate the framework on some computer vision tasks. Despite its significant potential, this project investigates a relatively less explored area of study in computer vision and machine learning. The goal of the project is to generate preliminary results to demonstrate its feasibility and to identify areas of improvements.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.