Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space
Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space
复制标题
DOI：
10.48550/arxiv.2206.11895
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Jinghuan Shang;Srijan Das;M. Ryoo
中科院分区：
文献类型：
--
作者：
Jinghuan Shang;Srijan Das;M. Ryoo
由于视觉皮层支持3D结构的感知，人类在理解视点变化方面非常灵活。相比之下，大多数从2D图像池中学习视觉表示的计算机视觉模型通常无法推广到新的相机视点。最近，视觉架构已经转向无卷积架构，视觉变换器，它对从图像补丁中派生的令牌进行操作。然而，这些变换器不执行显式操作来学习用于视觉理解的视点不可知表示。为此，我们提出了一个3D令牌表示层（3DTRL），估计视觉令牌的3D位置信息，并利用它来学习视点不可知的表示。3DTRL的关键元素包括一个伪深度估计器和一个学习的相机矩阵，用于对令牌进行几何变换，以无监督的方式进行训练。这些使得3DTRL能够从2D补丁恢复令牌的3D位置信息。实际上，3DTRL很容易插入到Transformer中。我们的实验证明了3DTRL在许多视觉任务中的有效性，包括图像分类，多视图视频对齐和动作识别。具有3DTRL的模型在所有任务中以最小的附加计算优于其骨干变压器。我们的代码可在https://github.com/elicassion/3DTRL上获得。
Humans are remarkably flexible in understanding viewpoint changes due to visual cortex supporting the perception of 3D structure. In contrast, most of the computer vision models that learn visual representation from a pool of 2D images often fail to generalize over novel camera viewpoints. Recently, the vision architectures have shifted towards convolution-free architectures, visual Transformers, which operate on tokens derived from image patches. However, these Transformers do not perform explicit operations to learn viewpoint-agnostic representation for visual understanding. To this end, we propose a 3D Token Representation Layer (3DTRL) that estimates the 3D positional information of the visual tokens and leverages it for learning viewpoint-agnostic representations. The key elements of 3DTRL include a pseudo-depth estimator and a learned camera matrix to impose geometric transformations on the tokens, trained in an unsupervised fashion. These enable 3DTRL to recover the 3D positional information of the tokens from 2D patches. In practice, 3DTRL is easily plugged-in into a Transformer. Our experiments demonstrate the effectiveness of 3DTRL in many vision tasks including image classification, multi-view video alignment, and action recognition. The models with 3DTRL outperform their backbone Transformers in all the tasks with minimal added computation. Our code is available at https://github.com/elicassion/3DTRL.