RI: Small: Toward Efficient and Robust Dynamic Scene Understanding Based on Visual Correspondences

寻找对应关系是计算机视觉中的一个基本问题;视觉对应关系为机器提供了有用的线索，使其能够以类似于人类的方式理解其动态环境。例如，当智能体四处移动时，它可能会了解到像山脉这样的遥远物体通常不会移动太多，而附近的建筑物和灌木丛似乎在环境中快速移动，因为智能体相对于它们改变位置。虽然在解决各种形式的视觉对应问题方面取得了重大进展，但不同的对应模型尽管具有内在的相似性，但仍保持不同的设计，使得有效的设计原则和学习的表示难以从一个问题转移到另一个问题。为了应对这一挑战，该项目旨在用统一的模型解决不同的视觉对应问题。在此过程中，该项目还将解决在具有不同视觉外观和严重资源限制的情景中实施所开发模型的两个实际方面。这些进展有望解锁新的应用程序，并提高增强现实，体育广播，体育分析，机器人等领域的动态场景理解，项目成果还可能通过增强用户日常生活中的认知和身体能力的解决方案揭开新市场和经济机遇。研究人员团队将积极将拟议的研究纳入课程开发，并吸引本科研究人员参与该项目。该项目特别适合推广活动，以扩大代表性不足和K-12学生的参与，通过连接抽象的技术概念与有形的研究演示。该项目有三个紧密相连的推力，在对应关系的确定，在这些对应关系的应用，并在使这些算法的部署有效和强大的基本进展。具体而言，首先，将开发一个统一的模型来解决从2D到3D的所有视觉对应问题，利用Transformer模型的最新进展和从大规模未标记数据中进行的自监督学习。Transformer模型自然地捕获具有较少归纳偏差的候选者的对应关系，使其成为从大规模数据中学习的更好选择，并在从数据丰富的域转移时提高数据贫乏的域的准确性。其次，通过这些对应关系，将解锁新的应用程序，以推进动态场景理解，特别是慢动作视频合成和机器人避障。最后，研究人员将研究在边缘计算设备上部署模型时提高效率和鲁棒性的机制。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Finding correspondences is a fundamental problem in computer vision; visual correspondences provide useful cues for a machine to understand its dynamic surroundings in a manner similar to what humans do. For instance, as an agent moves around, it may learn that objects that are far away like mountains typically do not move much, whereas nearby buildings and bushes appear to move rapidly in the environments as the agent changes position relative to them. Although significant advances have been made in solving various forms of visual correspondence problems, different correspondence models maintain different designs despite their inherent similarity, making the effective design principles and the learned representations difficult to transfer from one problem to another. In response to this challenge, this project aims to solve disparate visual correspondence problems with a unified model. In doing so, the project will also address two practical aspects of implementation of the developed models in scenarios with diverse visual appearance and significant resource constraints. These advances are expected to unlock novel applications and improve dynamic scene understanding in the areas of Augmented Reality, sports broadcasting, sports analytics, robotics, etc. The project outcomes may also unveil new markets and economic opportunities through solutions that augment cognitive and physical abilities of users in their daily lives. The team of researchers will actively integrate proposed research into the curriculum development and attract undergraduate researchers to the project. This project is particularly well-suited for outreach activities to broaden participation of underrepresented and K-12 students, by connecting abstract technical concepts with tangible research demonstrations.The project has three tightly connected thrusts, presenting fundamental advances in correspondence determination, in applications of these correspondences, and in making these algorithms efficient and robust in deployment. Concretely, first, a unified model to solve all the visual correspondence problems, ranging from 2D to 3D, will be developed, taking advantage of recent progress of the Transformer model and self-supervised learning from large-scale unlabeled data. The Transformer model naturally captures the correspondences of candidates with less inductive bias, making it a better choice to learn from the large-scale data and improve accuracy of data-poor domains when transferred from data-rich ones. Second, with the correspondences, novel applications will be unlocked to advance dynamic scene understanding, particularly for slow-motion video synthesis and robotic obstacle avoidance. Finally, the investigators will study mechanisms to improve efficiency and robustness when deploying the models on edge computing devices. The developed algorithms will be rigorously evaluated on standard benchmarks and in real-world deployment on edge devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.