RI: Small: Toward Efficient and Robust Dynamic Scene Understanding Based on Visual Correspondences

寻找对应关系是计算机视觉中的一个基本问题；视觉对应为机器以类似于人类的方式理解其动态环境提供了有用的线索。例如，当智能体四处移动时，它可能会了解到远处的物体（如山脉）通常不会移动太多，而附近的建筑物和灌木丛在环境中移动得很快，因为智能体改变了相对于它们的位置。尽管在解决各种形式的视觉对应问题方面取得了重大进展，但不同的对应模型保持不同的设计，尽管它们具有内在的相似性，这使得有效的设计原则和学习到的表示难以从一个问题转移到另一个问题。为了应对这一挑战，该项目旨在用统一的模型解决不同的视觉对应问题。在这样做的过程中，该项目还将处理在具有不同视觉外观和重要资源约束的场景中实现已开发模型的两个实际方面。这些进步有望在增强现实、体育广播、体育分析、机器人等领域开启新的应用，提高对动态场景的理解。项目成果还可能通过解决方案揭示新的市场和经济机会，增强用户在日常生活中的认知和身体能力。研究团队将积极将建议的研究纳入课程发展，并吸引本科生研究人员参与项目。通过将抽象的技术概念与具体的研究演示联系起来，该项目特别适合于扩大代表性不足和K-12学生参与的外展活动。该项目有三个紧密相连的重点，展示了通信确定、这些通信的应用以及使这些算法在部署中高效和健壮方面的基本进展。具体而言，首先，将利用Transformer模型的最新进展和大规模未标记数据的自监督学习，开发一个统一的模型来解决从2D到3D的所有视觉对应问题。Transformer模型自然地以较少的归纳偏差捕获候选对象的对应关系，使其成为从大规模数据中学习的更好选择，并在从数据丰富的领域转移时提高数据贫乏领域的准确性。其次，随着通信的发展，新的应用将被解锁，以推进动态场景的理解，特别是慢动作视频合成和机器人避障。最后，研究人员将研究在边缘计算设备上部署模型时提高效率和鲁棒性的机制。开发的算法将在标准基准和实际部署的边缘设备上进行严格评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Finding correspondences is a fundamental problem in computer vision; visual correspondences provide useful cues for a machine to understand its dynamic surroundings in a manner similar to what humans do. For instance, as an agent moves around, it may learn that objects that are far away like mountains typically do not move much, whereas nearby buildings and bushes appear to move rapidly in the environments as the agent changes position relative to them. Although significant advances have been made in solving various forms of visual correspondence problems, different correspondence models maintain different designs despite their inherent similarity, making the effective design principles and the learned representations difficult to transfer from one problem to another. In response to this challenge, this project aims to solve disparate visual correspondence problems with a unified model. In doing so, the project will also address two practical aspects of implementation of the developed models in scenarios with diverse visual appearance and significant resource constraints. These advances are expected to unlock novel applications and improve dynamic scene understanding in the areas of Augmented Reality, sports broadcasting, sports analytics, robotics, etc. The project outcomes may also unveil new markets and economic opportunities through solutions that augment cognitive and physical abilities of users in their daily lives. The team of researchers will actively integrate proposed research into the curriculum development and attract undergraduate researchers to the project. This project is particularly well-suited for outreach activities to broaden participation of underrepresented and K-12 students, by connecting abstract technical concepts with tangible research demonstrations.The project has three tightly connected thrusts, presenting fundamental advances in correspondence determination, in applications of these correspondences, and in making these algorithms efficient and robust in deployment. Concretely, first, a unified model to solve all the visual correspondence problems, ranging from 2D to 3D, will be developed, taking advantage of recent progress of the Transformer model and self-supervised learning from large-scale unlabeled data. The Transformer model naturally captures the correspondences of candidates with less inductive bias, making it a better choice to learn from the large-scale data and improve accuracy of data-poor domains when transferred from data-rich ones. Second, with the correspondences, novel applications will be unlocked to advance dynamic scene understanding, particularly for slow-motion video synthesis and robotic obstacle avoidance. Finally, the investigators will study mechanisms to improve efficiency and robustness when deploying the models on edge computing devices. The developed algorithms will be rigorously evaluated on standard benchmarks and in real-world deployment on edge devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.