权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

RI: Small: Learning Dynamics and Evolution towards Cognitive Understanding of Videos

RI：小：视频认知理解的学习动态和演化

基本信息

批准号：
1813709
负责人：
Chenliang Xu
金额：
$ 45万
依托单位：
University of Rochester
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-09-01 至 2021-08-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1813709&HistoricalAwards=false
关键词：
RI Small Learning Dynamics Evolution

项目摘要

A fundamental capability of human intelligence is being able to learn to act by watching instructional videos. Such capability is reflected in abstraction and summarization of the instructional procedures as well as in answering questions such as "why" and "how" something happened in the video. This project aims to build computational models that are able to perform well in above tasks, which require, beyond the conventional recognition of objects, actions and attributes in the scene, the higher-order inference of any relations therein. Here, the higher-order inference refers to inference that cannot be answered immediately by direct observations and thus requires stronger semantics. The developed technology will enable many applications in other fields, e.g., multimedia (video indexing and retrieval), robotics (reasoning capability of why and how questions), and healthcare (assistive devices for visually impaired people). In addition, the project will contribute to education and diversity by involving underrepresented groups in research activities, integrating research results into teaching curriculum, and conducting outreach activities to local K-12 communities. The research will develop a framework to perform higher-order inference in understanding web instructional videos, such that models devised in this framework are capable of not only discovering and captioning procedures that constitute the instructional event but also answering questions such as why and how something happened. The framework is built on a video story graph that models the dynamics (the composition of actions at different scales) and evolution (the change in object states and attributes), and it supports higher-order inference upon deep learning units and incorporation of external knowledge graph in a unified framework. Methodologies to extract such video story graphs and use them to discover, caption procedures and perform question-answering will be explored. Expected outcomes of this project include: a software package for constructing and performing inference on video story graphs and incorporating external knowledge; a web-deployed system to process user-uploaded instructional videos; and a large video dataset with procedure and question-answering annotations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人类智能的一个基本能力是能够通过观看教学视频来学习行动。这种能力体现在对教学过程的抽象和概括，以及回答视频中“为什么”和“如何”发生的问题。该项目旨在构建能够在上述任务中表现良好的计算模型，这些任务需要超越场景中对象，动作和属性的传统识别，其中任何关系的高阶推理。这里，高阶推理指的是不能通过直接观察立即回答的推理，因此需要更强的语义。开发的技术将使许多应用在其他领域，例如，多媒体（视频索引和检索）、机器人（为什么和如何问题的推理能力）和医疗保健（视障人士的辅助设备）。此外，该项目将通过让代表性不足的群体参与研究活动、将研究成果纳入教学课程以及向当地K-12社区开展外联活动，促进教育和多样性。该研究将开发一个框架，以执行高阶推理理解网络教学视频，这样，在这个框架中设计的模型不仅能够发现和字幕程序，构成教学事件，但也回答问题，如为什么和如何发生的事情。该框架建立在一个视频故事图上，该图对动态（不同尺度下的动作组合）和演化（对象状态和属性的变化）进行建模，它支持对深度学习单元的高阶推理，并将外部知识图纳入统一的框架中。将探讨提取这种视频故事图并使用它们来发现、说明程序和回答问题的方法。这一项目的预期成果包括：一个软件包，用于对视频故事图进行构建和推理，并纳入外部知识;一个网络部署系统，用于处理用户上传的教学视频;和一个大的视频数据集的过程和问题-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响进行评估，被认为值得支持审查标准。

项目成果

期刊论文数量（45）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Audio-Visual Event Localization in the Wild

野外视听事件定位

DOI：
发表时间：
2019
期刊：
IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops
影响因子：
0
作者：
Tian, Yapeng;Shi, Jing;Li, Bochen;Duan, Zhiyao;Xu, Chenliang
通讯作者：
Xu, Chenliang

Video Re-localization via Cross Gated Bilinear Matching

通过交叉门双线性匹配进行视频重新定位