RI: Small: Toward Efficient and Robust Dynamic Scene Understanding Based on Visual Correspondences

寻找对应关系是计算机视觉中的一个基本问题；视觉对应关系为机器提供有用的线索，以类似于人类所做的方式理解其动态环境。例如，当代理四处移动时，它可能会了解到，像山脉这样遥远的对象通常不会移动太多，而附近的建筑物和灌木丛似乎在环境中快速移动，因为代理相对于它们的位置发生了变化。尽管在解决各种形式的视觉对应问题方面取得了重大进展，但不同的对应模型保持着不同的设计，尽管它们固有的相似之处，使得有效的设计原则和学习的表征很难从一个问题转移到另一个问题。为了应对这一挑战，该项目旨在用一个统一的模型来解决不同的视觉通信问题。在这样做的同时，该项目还将解决在视觉外观多样和资源严重受限的情况下实施所开发模型的两个实际方面。这些进展预计将在增强现实、体育广播、体育分析、机器人等领域开启新的应用并提高对动态场景的理解。项目成果还可能通过增强用户日常生活中认知和身体能力的解决方案揭示新的市场和经济机会。研究人员团队将积极将拟议的研究纳入课程开发，并吸引本科生研究人员参与该项目。通过将抽象的技术概念与具体的研究演示联系起来，该项目特别适合于扩大代表不足和K-12学生参与的外联活动。该项目有三个紧密相连的推动力，展示了在对应关系确定、这些对应关系的应用以及使这些算法在部署中高效和健壮方面的基本进展。具体地说，首先，利用Transformer模型和大规模无标记数据的自监督学习的最新进展，将开发一个统一的模型来解决从2D到3D的所有视觉对应问题。Transformer模型以较少的归纳偏差自然地捕获候选对象的对应关系，使其成为从大规模数据中学习的更好选择，并在从数据丰富的领域转移时提高数据贫乏领域的准确性。其次，有了这些通信，新的应用将被解锁，以促进动态场景理解，特别是在慢动作视频合成和机器人避障方面。最后，研究人员将研究在边缘计算设备上部署模型时提高效率和稳健性的机制。开发的算法将根据标准基准和在边缘设备上的实际部署进行严格评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Finding correspondences is a fundamental problem in computer vision; visual correspondences provide useful cues for a machine to understand its dynamic surroundings in a manner similar to what humans do. For instance, as an agent moves around, it may learn that objects that are far away like mountains typically do not move much, whereas nearby buildings and bushes appear to move rapidly in the environments as the agent changes position relative to them. Although significant advances have been made in solving various forms of visual correspondence problems, different correspondence models maintain different designs despite their inherent similarity, making the effective design principles and the learned representations difficult to transfer from one problem to another. In response to this challenge, this project aims to solve disparate visual correspondence problems with a unified model. In doing so, the project will also address two practical aspects of implementation of the developed models in scenarios with diverse visual appearance and significant resource constraints. These advances are expected to unlock novel applications and improve dynamic scene understanding in the areas of Augmented Reality, sports broadcasting, sports analytics, robotics, etc. The project outcomes may also unveil new markets and economic opportunities through solutions that augment cognitive and physical abilities of users in their daily lives. The team of researchers will actively integrate proposed research into the curriculum development and attract undergraduate researchers to the project. This project is particularly well-suited for outreach activities to broaden participation of underrepresented and K-12 students, by connecting abstract technical concepts with tangible research demonstrations.The project has three tightly connected thrusts, presenting fundamental advances in correspondence determination, in applications of these correspondences, and in making these algorithms efficient and robust in deployment. Concretely, first, a unified model to solve all the visual correspondence problems, ranging from 2D to 3D, will be developed, taking advantage of recent progress of the Transformer model and self-supervised learning from large-scale unlabeled data. The Transformer model naturally captures the correspondences of candidates with less inductive bias, making it a better choice to learn from the large-scale data and improve accuracy of data-poor domains when transferred from data-rich ones. Second, with the correspondences, novel applications will be unlocked to advance dynamic scene understanding, particularly for slow-motion video synthesis and robotic obstacle avoidance. Finally, the investigators will study mechanisms to improve efficiency and robustness when deploying the models on edge computing devices. The developed algorithms will be rigorously evaluated on standard benchmarks and in real-world deployment on edge devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.