Collaborative Research: RI:Medium:Understanding Events from Streaming Video - Joint Deep and Graph Representations, Commonsense Priors, and Predictive Learning

协作研究:RI:Medium:理解流视频中的事件 - 联合深度和图形表示、常识先验和预测学习

基本信息

  • 批准号:
    2348689
  • 负责人:
  • 金额:
    $ 28.51万
  • 依托单位:
  • 依托单位国家:
    美国
  • 项目类别:
    Continuing Grant
  • 财政年份:
    2023
  • 资助国家:
    美国
  • 起止时间:
    2023-10-01 至 2024-10-31
  • 项目状态:
    已结题

项目摘要

While it is easy for humans to process video data and extract meanings from it, it is extremely hard to design algorithms to do so. When developed, there are many applications of this technology, such as building assistive robotics or constructing smart spaces for independent living or monitoring wildlife. Video-data capture events, which are central to the content of human experience. Events consist of objects/people (who), location (where), time (when), actions (what), activities (how), and intent (why). This project develops a computer vision-based event understanding algorithm that operates in a self-supervised, streaming fashion. The algorithm will predict and detect old and new events, learn to build hierarchical event representations, all in the context of a prior knowledge-base that is updated over time. The intent is to generate interpretations of an event that go beyond what is seen, rather than just recognition. This research pushes the frontier of computer vision by coupling the self-supervised learning process with prior knowledge, moving the field towards open-world algorithms, and needing little or no supervision. Furthermore, this project will focus on recruitment and retention of undergraduate women students through freshman and sophomore years, with attention towards underrepresented minority students at the three sites: University of South Florida, Florida State University, and Oklahoma State University.At the core of the approach is a hybrid representational hierarchy that includes both continuous representations and symbolic graph-based representations. The continuous-valued representation is the standard, vector-valued deep learning stack that ends in an embedding vector of some object or action concept in the knowledge base. The next level of the representation consists of elementary symbolic compositions of these verbs and nouns. These elementary compositions, when associated with concepts from a knowledge-base they makeup an event interpretation, containing descriptions that go beyond what is observed in the image. These symbolic levels are built using Grenander's canonical representations from pattern theory. These representations, which have flexible graph-structured backbones, are more expressive than other well-known graphical models. The specific technical aims of the project are four-fold. First, it seeks to integrate function-based continuous with energy-based Grenander's canonical symbolic representations from pattern theory into one integrated formulation based on equilibrium propagation. Second, it will research and develop ways to use and modify commonsense knowledge bases. This will help to go beyond the closed world assumption, which is implicit in the current practice of annotated data-based deep learning approaches. Third, it will develop dynamical models on graph manifolds, which will enable generative modeling of graph structures for prediction and discovery of new concepts. Fourth, inspired by finding from human perception experiments and neuroscience, it will design predictive self-supervised learning over both continuous and symbolic representations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
虽然人类很容易处理视频数据并从中提取意义,但设计算法来做到这一点却非常困难。当开发出来时,这项技术有许多应用,例如建造辅助机器人或建造独立生活或监测野生动物的智能空间。 视频数据捕获事件,这是人类经验内容的核心。事件包括对象/人(谁)、位置(哪里)、时间(何时)、动作(什么)、活动(如何)和意图(为什么)。该项目开发了一种基于计算机视觉的事件理解算法,该算法以自我监督的流式方式运行。该算法将预测和检测旧事件和新事件,学习构建分层事件表示,所有这些都是在随时间更新的先验知识库的背景下进行的。其目的是产生对事件的解释,超越所看到的,而不仅仅是认识。 这项研究通过将自监督学习过程与先验知识相结合,将该领域推向开放世界算法,并且几乎不需要监督,从而推动了计算机视觉的前沿。此外,该项目将侧重于招收和保留大学一年级和二年级的女本科生,同时关注三个地点代表性不足的少数民族学生:南佛罗里达大学、佛罗里达州立大学和俄克拉荷马州州立大学。连续值表示是标准的向量值深度学习堆栈,以知识库中某个对象或动作概念的嵌入向量结束。下一个层次的表征由这些动词和名词的基本符号组成。当这些基本成分与来自知识库的概念相关联时,它们构成事件解释,包含超出图像中观察到的描述。这些符号层次是使用Grenander的模式理论的规范表示构建的。这些表示具有灵活的图形结构的主干,比其他知名的图形模型更具表现力。该项目的具体技术目标有四个方面。首先,它旨在整合功能为基础的连续与能量为基础的Grenander的规范符号表示从模式理论到一个综合的配方平衡传播的基础上。其次,它将研究和开发如何使用和修改常识知识库。这将有助于超越封闭世界的假设,这在当前基于注释数据的深度学习方法的实践中是隐含的。第三,它将开发图形流形上的动态模型,这将使图形结构的生成建模能够预测和发现新概念。 第四,受人类感知实验和神经科学发现的启发,该奖项将设计出连续和符号表示的预测性自我监督学习。该奖项反映了NSF的法定使命,并通过使用基金会的智力价值和更广泛的影响审查标准进行评估,被认为值得支持。

项目成果

期刊论文数量(1)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
IS-GGT: Iterative Scene Graph Generation with Generative Transformers
{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

Sathyanarayanan Aakur其他文献

Enhancing corn yield prediction: Optimizing data quality or model complexity?
提高玉米产量预测:优化数据质量还是模型复杂度?
  • DOI:
    10.1016/j.atech.2024.100671
  • 发表时间:
    2024-12-01
  • 期刊:
  • 影响因子:
    5.700
  • 作者:
    Yuting Zhou;Shengfang Ma;Huihui Zhang;Sathyanarayanan Aakur
  • 通讯作者:
    Sathyanarayanan Aakur

Sathyanarayanan Aakur的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

{{ truncateString('Sathyanarayanan Aakur', 18)}}的其他基金

CAREER:Towards Causal Multi-Modal Understanding with Event Partonomy and Active Perception
职业:通过事件部分和主动感知实现因果多模态理解
  • 批准号:
    2348690
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Continuing Grant
CAREER:Towards Causal Multi-Modal Understanding with Event Partonomy and Active Perception
职业:通过事件部分和主动感知实现因果多模态理解
  • 批准号:
    2143150
  • 财政年份:
    2022
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Continuing Grant
Collaborative Research: RI:Medium:Understanding Events from Streaming Video - Joint Deep and Graph Representations, Commonsense Priors, and Predictive Learning
协作研究:RI:Medium:理解流视频中的事件 - 联合深度和图形表示、常识先验和预测学习
  • 批准号:
    1955230
  • 财政年份:
    2020
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Continuing Grant

相似国自然基金

Research on Quantum Field Theory without a Lagrangian Description
  • 批准号:
    24ZR1403900
  • 批准年份:
    2024
  • 资助金额:
    0.0 万元
  • 项目类别:
    省市级项目
Cell Research
  • 批准号:
    31224802
  • 批准年份:
    2012
  • 资助金额:
    24.0 万元
  • 项目类别:
    专项基金项目
Cell Research
  • 批准号:
    31024804
  • 批准年份:
    2010
  • 资助金额:
    24.0 万元
  • 项目类别:
    专项基金项目
Cell Research (细胞研究)
  • 批准号:
    30824808
  • 批准年份:
    2008
  • 资助金额:
    24.0 万元
  • 项目类别:
    专项基金项目
Research on the Rapid Growth Mechanism of KDP Crystal
  • 批准号:
    10774081
  • 批准年份:
    2007
  • 资助金额:
    45.0 万元
  • 项目类别:
    面上项目

相似海外基金

Collaborative Research: RI: Medium: Principles for Optimization, Generalization, and Transferability via Deep Neural Collapse
合作研究:RI:中:通过深度神经崩溃实现优化、泛化和可迁移性的原理
  • 批准号:
    2312841
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: RI: Medium: Principles for Optimization, Generalization, and Transferability via Deep Neural Collapse
合作研究:RI:中:通过深度神经崩溃实现优化、泛化和可迁移性的原理
  • 批准号:
    2312842
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: RI: Small: Foundations of Few-Round Active Learning
协作研究:RI:小型:少轮主动学习的基础
  • 批准号:
    2313131
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: RI: Medium: Lie group representation learning for vision
协作研究:RI:中:视觉的李群表示学习
  • 批准号:
    2313151
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Continuing Grant
Collaborative Research: RI: Medium: Principles for Optimization, Generalization, and Transferability via Deep Neural Collapse
合作研究:RI:中:通过深度神经崩溃实现优化、泛化和可迁移性的原理
  • 批准号:
    2312840
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: RI: Small: Deep Constrained Learning for Power Systems
合作研究:RI:小型:电力系统的深度约束学习
  • 批准号:
    2345528
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: RI: Small: Motion Fields Understanding for Enhanced Long-Range Imaging
合作研究:RI:小型:增强远程成像的运动场理解
  • 批准号:
    2232298
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: CompCog: RI: Medium: Understanding human planning through AI-assisted analysis of a massive chess dataset
合作研究:CompCog:RI:中:通过人工智能辅助分析海量国际象棋数据集了解人类规划
  • 批准号:
    2312374
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: CompCog: RI: Medium: Understanding human planning through AI-assisted analysis of a massive chess dataset
合作研究:CompCog:RI:中:通过人工智能辅助分析海量国际象棋数据集了解人类规划
  • 批准号:
    2312373
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
Collaborative Research: RI: Small: End-to-end Learning of Fair and Explainable Schedules for Court Systems
合作研究:RI:小型:法院系统公平且可解释的时间表的端到端学习
  • 批准号:
    2232055
  • 财政年份:
    2023
  • 资助金额:
    $ 28.51万
  • 项目类别:
    Standard Grant
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了