RI: Small: Learning 3D Equivariant Visual Representation for Animals

计算机视觉的最新进展使得在野外以惊人的精度跟踪人类成为可能。然而，尽管这些新方法对生物学、神经科学和医学等多个学科产生了重大的科学和社会影响，但将这些新方法推广到不同的动物物种还为时过早。这些方法建立在监督学习范式的基础上，需要大量的注释数据，但获得动物物种的可比注释视觉数据从根本上是不可行的，因为它需要专业知识，而且物种特定图像的可用性有限，导致跟踪模型存在很大偏差。在这个研究项目中，研究者将开发新的计算机视觉理论和算法，可以有效地探索大量或潜在无限数量的未标记动物图像。开发的基本原理是通用的，因此很容易应用，经过一些修改，类似的计算机视觉任务，如2D/3D人体姿态估计，可变形对象注册和密集对应估计。该项目将研究与教育相结合，并通过一系列项目向代表性不足的K-12学生提供服务。虽然这个研究项目的主要重点是学习动物的视觉表示，但这个项目解决了一个核心的计算机视觉问题，即给定有限数量的标记数据的地标定位/关键点检测/姿态估计。该项目将利用三维等变性，即铰接可变形物体视觉数据的固有属性，揭示跨视图、时间和物种的共享和可重复的视觉关系。研究者将通过动物的三维重建将提出的等方差整合到表征学习中，这将促进视觉信息从一个图像到另一个图像的传递。更具体地说，该项目将致力于：(1)一个新的多视图几何来学习跨视图的视觉转换，它允许跨视图自我监督；(2)从运动中重新制定非刚性结构，通过3D姿态参数化，以便从单目视频中学习；(3)解开外观和三维姿态的纠缠，学习跨物种的视觉转换。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in computer vision make it possible to track humans in the wild with remarkable accuracy. Generalizing these new approaches towards diverse animal species, however, is still premature, despite the significant scientific and societal impact on multiple disciplines such as biology, neuroscience, and medicine. These approaches are built upon a supervised learning paradigm that requires sizable annotated data, but attaining comparable annotated visual data for animal species is fundamentally infeasible, as it requires expert knowledge and there is limited availability of species-specific images, leading to a large bias in the tracking models. In this research project, the investigator will develop new computer vision theories and algorithms that can effectively explore a large or potentially infinite number of unlabeled images of animals. The developed fundamentals are generic, and therefore readily applicable, with some modifications, to similar computer vision tasks, such as 2D/3D human pose estimation, deformable object registration, and dense correspondence estimation. The project integrates research with education and outreaches to K-12 students of under-represented groups through a series of programs.While the primary focus of this research program is on learning a visual representation of animals, this project addresses a core computer vision problem of landmark localization/keypoint detection/pose estimation given a limited amount of labeled data. The project will make use of 3D equivariance, an intrinsic property in visual data of articulated deformable objects, to uncover shared and repeatable visual relationships across views, time, and species. The investigator will integrate the proposed equivariance through 3D reconstruction of animals into representation learning, which will facilitate the transfer of visual information from one image to another. More specifically, the project will work on: (1) a new multiview geometry to learn the visual transformation across views, which allows cross-view self-supervision; (2) a re-formulation of non-rigid structure from motion, parametrized by 3D pose to enable learning from monocular videos; and (3) disentanglement of appearance and 3D pose to learn the visual transformation across species.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.