RI: Small: Learning 3D Equivariant Visual Representation for Animals

计算机视觉的最新进展使在野外以惊人的精度跟踪人类成为可能。然而，尽管这些新方法对生物学、神经科学和医学等多个学科产生了重大的科学和社会影响，但将这些新方法推广到不同的动物物种仍然为时过早。这些方法建立在监督学习范式的基础上，需要大量的注释数据，但获得动物物种的可比注释视觉数据从根本上是不可行的，因为它需要专业知识，而且物种特定图像的可获得性有限，导致跟踪模型存在很大偏差。在这个研究项目中，研究人员将开发新的计算机视觉理论和算法，可以有效地探索大量或潜在无限数量的未标记动物图像。开发的基本原理是通用的，因此经过一些修改很容易应用于类似的计算机视觉任务，例如2D/3D人体姿势估计、可变形对象配准和密集对应估计。该项目将研究与教育相结合，并通过一系列计划为代表不足的群体的K-12学生提供外展服务。虽然该研究计划的主要重点是学习动物的视觉表征，但该项目解决了在有限数量的标记数据下的地标定位/关键点检测/姿势估计的核心计算机视觉问题。该项目将利用3D等差，这是关节可变形物体视觉数据的内在属性，以揭示跨视图、时间和物种的共享和可重复的视觉关系。研究人员将通过动物的3D重建将所提出的等方差整合到表征学习中，这将促进视觉信息从一幅图像转移到另一幅图像。更具体地说，该项目将致力于：(1)新的多视角几何学，以学习跨视点的视觉转换，允许跨视点自我监督；(2)从运动中重新制定非刚性结构，通过3D姿势进行参数化，以能够从单目视频中学习；以及(3)外观和3D姿势的分离，以学习跨物种的视觉转换。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in computer vision make it possible to track humans in the wild with remarkable accuracy. Generalizing these new approaches towards diverse animal species, however, is still premature, despite the significant scientific and societal impact on multiple disciplines such as biology, neuroscience, and medicine. These approaches are built upon a supervised learning paradigm that requires sizable annotated data, but attaining comparable annotated visual data for animal species is fundamentally infeasible, as it requires expert knowledge and there is limited availability of species-specific images, leading to a large bias in the tracking models. In this research project, the investigator will develop new computer vision theories and algorithms that can effectively explore a large or potentially infinite number of unlabeled images of animals. The developed fundamentals are generic, and therefore readily applicable, with some modifications, to similar computer vision tasks, such as 2D/3D human pose estimation, deformable object registration, and dense correspondence estimation. The project integrates research with education and outreaches to K-12 students of under-represented groups through a series of programs.While the primary focus of this research program is on learning a visual representation of animals, this project addresses a core computer vision problem of landmark localization/keypoint detection/pose estimation given a limited amount of labeled data. The project will make use of 3D equivariance, an intrinsic property in visual data of articulated deformable objects, to uncover shared and repeatable visual relationships across views, time, and species. The investigator will integrate the proposed equivariance through 3D reconstruction of animals into representation learning, which will facilitate the transfer of visual information from one image to another. More specifically, the project will work on: (1) a new multiview geometry to learn the visual transformation across views, which allows cross-view self-supervision; (2) a re-formulation of non-rigid structure from motion, parametrized by 3D pose to enable learning from monocular videos; and (3) disentanglement of appearance and 3D pose to learn the visual transformation across species.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.