Understanding and Improving Deep Learning for Structured Data

深度学习在处理具有规则结构的数据方面取得了巨大的成功，如图像(网格)、语言(序列)和语音(序列)。然而，具有不规则结构的数据(如计算机视觉中的点云、机器人中的多智能体交互图、自然语言处理中的解析树、计算化学中的分子、生物学中的蛋白质结构以及计算社会科学中的社会网络)是无处不在的，并对现有的深度学习方法提出了挑战。在这个研究项目中，我们旨在了解当前针对结构化数据的深度学习模型的局限性，并相应地改进它们。在第一部分中，我们将研究图神经网络(GNN)和变压器在推理任务中的分布外(OOD)泛化能力。例如，在我们知道存在正确算法的算法推理任务中，我们将调查GNNS/Transformers是否可以学习概括到OOD图形数据的图形算法。这些研究将有助于设计具有更好感应偏向的新型模型。一旦我们从这些明确定义的任务中得到结论，我们就会转向更具挑战性和更现实的推理问题，例如，基于学习神经网络的NP-Hard问题的求解器，视觉推理任务，如视觉问题回答和场景图预测，以及自动驾驶的运动规划。该部分将有助于改进推理任务的深度学习模型，提高其对结构化数据的面向对象的泛化能力。在第二部分中，我们计划为结构化数据设计新的深度生成模型。鉴于最近的深度生成模型在图像上的优越性能，将其推广到像图这样的结构化数据是很有吸引力的。然而，我们需要克服两个主要挑战：1)为图建立可表达和排列不变的概率模型；2)为离散随机变量设计定制的采样和学习算法。此外，我们计划将无条件的深度生成模型推广到有条件的环境中，以便我们可以从数据中学习潜在的结构。这样的模型可以描述不确定性，并有助于提高可解释性，这对于3D计算视觉中的点云生成和药物发现中的分子生成等真实世界的应用至关重要。

Deep learning has achieved tremendous success in processing data with regular structures like images (grids), languages (sequences), and speech (sequences). However, data with irregular structure (e.g., point clouds in computer vision, multi-agent interaction graphs in robotics, parsing trees in natural language processing, molecules in computational chemistry, protein structure in biology, and social networks in computational social science) is ubiquitous and poses challenges to current deep learning methods. In this research program, we aim to understand the limitations of current deep learning models for structured data and improve them accordingly. In the first part, we will study the out-of-distribution (OOD) generalization ability of graph neural networks (GNNs) and Transformers in the context of reasoning tasks. For example, in algorithmic reasoning tasks where we know correct algorithms exist, we will investigate if GNNs/Transformers can learn graph algorithms that generalize to OOD graph data. These studies would shed light on designing novel models with better inductive bias. Once we obtain conclusions from such well-defined tasks, we will move to more challenging and realistic reasoning problems, e.g., learning neural networks based solvers for NP-hard problems, visual reasoning tasks such as visual question answering and scene graph prediction, and motion planning for self-driving. This part would help improve deep learning models on reasoning tasks and improve their OOD generalization on structured data. In the second part, we plan to design novel deep generative models for structured data. Given the superior performance of the recent deep generative models on images, it is appealing to generalize them to structured data like graphs. However, we need to overcome two main challenges: 1) building expressive and permutation-invariant probabilistic models for graphs; 2) designing customized sampling and learning algorithms for discrete random variables. Moreover, we plan to extend unconditional deep generative models to the conditional setting so that we can learn latent structures from data. Such models could describe the uncertainty and help improve the interpretability, which are crucial for real-world applications like point clouds generation in 3D compute vision and molecule generation for drug discovery.