CAREER: A Unifying Stochastic Framework for Temporally Consistent Computer Vision Models

序列蒙特卡罗方法是一种有效的机制来整合随时间变化的观察在计算机视觉问题中，特别是与深度神经网络产生的特征相关联。然而，目前尚不清楚如何将这些网络解释为随机推理系统的组成部分。该项目将时序蒙特卡罗方法与神经网络相结合，为计算机视觉任务创建一个可训练的随机框架。开发的框架将使自主和机器人系统的设计能够解释其环境并与之交互，这是在复杂、不受约束的情况下必须执行的任务自动化的关键组件。这些能力将通过新型农业机器人系统的开发来证明，该系统可以在不同的空间和时间粒度水平上生成精确的农作物模型。该项目将特别关注代表性不足的人群，为研究生和本科生提供有关人工智能主题及其在农业问题上的应用的研究机会和实践培训。它还将为学生提供基本的创业技能，使他们能够识别可以使用机器学习和人工智能方法解决的广泛社会相关问题。本研究将创建一个随机框架，以端到端的方式学习如何利用有关感兴趣对象的语义信息来吸收空间和时间视觉信息，并在计算机视觉算法中强制时间一致性。将多目标分割和跟踪问题作为非参数像素概率分布估计任务，将使设计不确定性感知模型成为可能，该模型可以了解给定对象周围环境下对象的外观如何随时间变化。这些研究工作还将为运动模型的表示引入一种新的范式，该范式在像素级上强制视频帧之间的时间一致性，从而避免了对目标检测和定位技术的需要。最后，这些时间关联方法将大大简化非结构化环境中复杂对象的识别和重建问题。通过整合与农业问题相关的参数，并扩展概率模型以满足特定领域的约束，该项目将设计新的技术来提取语义信息，并在单个叶子的粒度上生成整个果园的大规模重建。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Sequential Monte Carlo methods are an effective mechanism to integrate observations over time in computer vision problems, especially in association with features generated by deep neural networks. However, it is still unclear how such networks can be interpreted as components of a stochastic inference system. This project will combine sequential Monte Carlo methods with neural networks to create a trainable stochastic framework for computer vision tasks. The developed framework will enable the design of autonomous and robotic systems that can interpret and interact with their environment, a critical component for the automation of tasks that must be performed in complex, unconstrained scenarios. These capabilities will be demonstrated through the development of novel agricultural robotic systems that generate accurate models of agricultural crops at varying levels of spatial and temporal granularity. With particular focus on under-represented populations, the project will provide research opportunities and hands-on training to graduate and undergraduate students on artificial intelligence topics and their applications to agricultural problems. It will also provide the students with foundational entrepreneurial skills that will allow them to identify problems of broad societal relevance that can be solved using machine learning and artificial intelligence methods.This research will create a stochastic framework that learns in an end-to-end manner how to leverage semantic information about objects of interest to assimilate spatial and temporal visual information and enforce temporal consistency in computer vision algorithms. Casting the multiple-object segmentation and tracking problem as a non-parametric pixel probability distribution estimation task will make it possible to devise uncertainty-aware models that learn how the appearance of objects varies over time given the context surrounding them. These research efforts will also introduce a new paradigm for the representation of motion models that enforce temporal consistency among video frames at the pixel level, obviating the need for object detection and localization techniques. Finally, these temporal association methods will substantially simplify the problem of recognizing and reconstructing complex objects in unstructured environments. By incorporating parameters of relevance to agricultural problems and extending the probabilistic models to satisfy domain-specific constraints, this project will devise novel techniques to extract semantic information and generate large-scale reconstructions of entire orchards at the granularity of individual leaves.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.