Shape Perception in Computer Vision

考虑建模一个杯子的任务，以便计算机视觉系统可以识别放置在相机前的任何杯子。我们如何根据杯子的各个部分来建模？我们如何从图像中恢复这些部分？我们如何确定一个我们从未见过的不寻常的物体是否可以用作杯子？我们如何利用深度学习的力量来完成这些任务？最后，我们卓越的人类视觉系统如何为这些深度学习解决方案的设计提供信息？揭示形状感知中的这些基本问题可能会对诸如物体表征、识别和操作等下游任务产生重大影响。在我们最近的工作中，我们证明了局部对称在人类形状感知中发挥的重要作用，并表明当输入增加局部对称信息时，标准深度学习架构的性能可以得到改善，即框架不能自己计算这些信息。我们以两种重要的方式扩展这项工作：1)我们将开发一个可以计算这些有价值信息的框架；2)我们将把这个模块嵌入到一个更大的端到端深度学习框架中，用于识别任务，鼓励更模块化的识别管道，更易于解释，并尊重对称在人类视觉中发挥的重要作用。我们的第二个项目解决了三维形状表征学习的问题。具体来说，对于给定的一组属于一个类别的物体的3d图像，我们如何学习该类物体的表示，该表示可用于生成/识别属于该类的新示例？在我们之前的工作中，我们学会了梳理或“解开”学习表征，以区分由于发音引起的对象的外在变化（例如，特定老虎腿的不同配置）和由于类内变化引起的对象的内在变化（例如，大型猫科动物家族的形状变化）。我们将在表示学习中扩展这项工作，根据物体的自然部分结构来学习物体的组成模型，这是一个深深植根于人类和计算机视觉的概念。在我们的第三个项目中，我们超越了学习纯粹的几何形状表征，转而学习更抽象的表征，考虑到物体的使用方式，即其可视性。回到我们的杯子的例子，一个锅可以被当作杯子使用，但在任何现代识别系统中都不会被归类为杯子。并不是杯子的精确几何形状提供了喝水的任务，而是这个物体既提供了容器，又提供了倾斜容器倒液体的把手。我们扩展了我们之前在学习形状和功能之间的映射方面的工作，以利用基于物理的模拟器和新形状表示的力量，以学习如何抓住物体以促进特定任务。

Consider the task of modeling a cup, so that a computer vison system can recognize any cup placed in front of a camera. How do we model the cup in terms of its parts? How do we recover those parts from an image? How do we determine if an unusual object we've never seen before can be used as a cup? How do we utilize the power of deep learning to perform these tasks? And finally, how can our remarkable human vision system inform the design of these deep learning solutions? Shedding light on these fundamental questions in shape perception could have a major impact on such downstream tasks as object representation, recognition, and manipulation.  In our recent work, we demonstrated the important role that local symmetry plays in human shape perception, and showed that the performance of a standard deep learning architecture can be improved when the input is augmented with local symmetry information, i.e., the framework cannot compute this information on its own. We extend that work in two important ways: 1) we will develop a framework that can compute this valuable information; and 2) we will embed this module in a larger end-to-end, deep learning framework for recognition tasks, encouraging a more modular recognition pipeline that's more interpretable, and respects the important role that symmetry plays in human vision. Our second project addresses the problem of 3-D shape representation learning. Specifically, for a given a set of 3-D images of objects belonging to a category, how do we learn a representation for that class of objects that can be used to generate/recognize new examples belonging to the class? In our previous work, we learned to tease apart, or "disentangle", the learned representation to differentiate between extrinsic variations in the object due to articulation (e.g., the different configurations of a particular tiger's legs) and intrinsic variations in the object due to within-class variation (e.g., the variations in shape across the family of big cats). We will extend that work in representation learning to learn compositional models of objects in terms of their natural part structure, a concept deeply rooted in both human and conputer vision. In our third project, we move beyond learning purely geometric representations of shape to learn more abstract representations that take into account how the object will be used, i.e., its affordances. Returning to our cup example, a pot could be used as a cup, but would never be categorized as a cup in any modern-day recognition system. It's not the precise geometry of the cup that affords the task of drinking, but the fact that the object affords both containment and a handle with which to tilt the container to pour the liquid. We extend our previous work in learning the mapping between shape and affordance to leverage the power of physics-based simulators and new  shape representations in order to learn how to grasp an object in order to facilitate a particular task.