CAREER: A Unifying Stochastic Framework for Temporally Consistent Computer Vision Models

顺序蒙特卡罗方法是一种有效的机制，可以在计算机视觉问题中随着时间的推移整合观察结果，特别是与深度神经网络生成的特征相关联。然而，目前还不清楚如何将这种网络解释为随机推理系统的组成部分。 该项目将联合收割机序列蒙特卡罗方法与神经网络相结合，为计算机视觉任务创建一个可训练的随机框架。开发的框架将使自主和机器人系统的设计，可以解释和与他们的环境，一个关键的组成部分，必须在复杂的，不受约束的情况下执行的任务自动化。这些能力将通过开发新型农业机器人系统来展示，这些系统可以在不同的空间和时间粒度水平上生成农作物的精确模型。该项目特别关注代表性不足的人群，将为研究生和本科生提供人工智能主题及其在农业问题中的应用的研究机会和实践培训。它还将为学生提供基本的创业技能，使他们能够识别可以使用机器学习和人工智能方法解决的广泛社会相关性问题。如何利用感兴趣对象的语义信息来同化空间和时间视觉信息，并在计算机中实现时间一致性视觉算法将多对象分割和跟踪问题转换为非参数像素概率分布估计任务，将使设计不确定性感知模型成为可能，该模型可以在给定对象周围的上下文的情况下学习对象的外观如何随时间变化。这些研究工作还将引入一种新的模式，用于表示运动模型，该模型在像素级上执行视频帧之间的时间一致性，从而避免了对对象检测和定位技术的需要。最后，这些时间关联方法将大大简化在非结构化环境中识别和重建复杂对象的问题。通过纳入与农业问题相关的参数并扩展概率模型以满足特定领域的约束，这个项目将设计新的技术来提取语义信息，并生成大量的，该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响进行评估，被认为值得支持审查标准。

Sequential Monte Carlo methods are an effective mechanism to integrate observations over time in computer vision problems, especially in association with features generated by deep neural networks. However, it is still unclear how such networks can be interpreted as components of a stochastic inference system. This project will combine sequential Monte Carlo methods with neural networks to create a trainable stochastic framework for computer vision tasks. The developed framework will enable the design of autonomous and robotic systems that can interpret and interact with their environment, a critical component for the automation of tasks that must be performed in complex, unconstrained scenarios. These capabilities will be demonstrated through the development of novel agricultural robotic systems that generate accurate models of agricultural crops at varying levels of spatial and temporal granularity. With particular focus on under-represented populations, the project will provide research opportunities and hands-on training to graduate and undergraduate students on artificial intelligence topics and their applications to agricultural problems. It will also provide the students with foundational entrepreneurial skills that will allow them to identify problems of broad societal relevance that can be solved using machine learning and artificial intelligence methods.This research will create a stochastic framework that learns in an end-to-end manner how to leverage semantic information about objects of interest to assimilate spatial and temporal visual information and enforce temporal consistency in computer vision algorithms. Casting the multiple-object segmentation and tracking problem as a non-parametric pixel probability distribution estimation task will make it possible to devise uncertainty-aware models that learn how the appearance of objects varies over time given the context surrounding them. These research efforts will also introduce a new paradigm for the representation of motion models that enforce temporal consistency among video frames at the pixel level, obviating the need for object detection and localization techniques. Finally, these temporal association methods will substantially simplify the problem of recognizing and reconstructing complex objects in unstructured environments. By incorporating parameters of relevance to agricultural problems and extending the probabilistic models to satisfy domain-specific constraints, this project will devise novel techniques to extract semantic information and generate large-scale reconstructions of entire orchards at the granularity of individual leaves.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.