权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

RI:Small:Collaborative Research: Understanding Human-Object Interactions from First-person and Third-person Videos

RI:Small：协作研究：从第一人称和第三人称视频中理解人与物体的交互

基本信息

批准号：
2104404
负责人：
Michael Ryoo
金额：
$ 22.84万
依托单位：
SUNY at Stony Brook
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-09-01 至 2022-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2104404&HistoricalAwards=false
关键词：
RI Small Collaborative Research Understanding

项目摘要

Ubiquitous cameras, together with ever increasing computing resources, are dramatically changing the nature of visual data and their analysis. Cities are adopting networked camera systems for policing and intelligent resource allocation, and individuals are recording their lives using wearable devices. For these camera systems to become truly smart and useful for people, it is crucial that they understand interesting objects in the scene and detect ongoing activities/events, while jointly considering continuous 24/7 videos from multiple sources. Such object-level and activity-level awareness in hospitals, elderly homes, and public places would provide assistive and quality-of-life technology for disabled and elderly people, provide intelligent surveillance systems to prevent crimes, and allow smart usage of environmental resources. This project will investigate novel computer vision algorithms that combine 1st-person videos (from wearable cameras) and 3rd-person videos (from static environmental cameras) for joint recognition of humans, objects, and their interactions. The key idea is to combine the two views' complementary and unique advantages for joint visual scene understanding. To this end, it will create a new dataset, and develop new algorithms that learn to recognize objects jointly across the views, learn human-object and human-human relationships through the two views, and anonymize the videos to preserve users' privacies. The project will provide new algorithms that have the potential to benefit applications in smart environments, security, and quality-of-life assistive technologies. The project will also perform complementary educational and outreach activities that engage students in research and STEM.This project will develop novel algorithms that learn from joint 1st-person videos (from wearable cameras) and 3rd-person videos (from static environmental cameras) for joint recognition of humans, objects, and their interactions. The 1st-person view is ideal for object recognition, while the 3rd-person view is ideal for human activity recognition. Thus, this project will investigate unique solutions to challenging problems that would otherwise be difficult to overcome when analyzing each viewpoint in isolation. The main research directions will be: (1) creating a benchmark 1st-person and 3rd-person video dataset to investigate this new problem; and developing algorithms that (2) learn to establish object and human correspondences between the two views; (3) learn object-action relationships across the views; and (4) anonymize the visual data for privacy-preserving visual recognition.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

无处不在的摄像头，加上不断增加的计算资源，正在戏剧性地改变视觉数据及其分析的性质。城市正在采用联网摄像头系统来进行治安和智能资源分配，个人正在使用可穿戴设备记录他们的生活。要让这些摄像系统变得真正智能和对人们有用，它们必须了解场景中感兴趣的对象，并检测正在进行的活动/事件，同时联合考虑来自多个来源的全天候连续视频。医院、养老院和公共场所的这种对象级别和活动级别的意识将为残疾人和老年人提供辅助和生活质量技术，提供智能监控系统以防止犯罪，并允许智能使用环境资源。该项目将研究新的计算机视觉算法，该算法将第一人称视频(来自可穿戴式摄像头)和第三人称视频(来自静态环境摄像头)相结合，以联合识别人、物体及其交互。其核心思想是将两个视点互补而独特的优势结合起来，共同进行视觉场景理解。为此，它将创建一个新的数据集，并开发新的算法，学习跨视图联合识别对象，通过两个视图学习人与对象和人与人的关系，并对视频进行匿名化以保护用户的隐私。该项目将提供新的算法，这些算法有可能使智能环境、安全和生活质量辅助技术中的应用受益。该项目还将开展互补的教育和推广活动，让学生参与研究和STEM。该项目将开发新的算法，从联合第一人称视频(来自可穿戴摄像头)和第三人称视频(来自静态环境摄像头)中学习，用于联合识别人、物体及其交互。第一人称视图适用于对象识别，而第三人称视图适用于人类活动识别。因此，这个项目将调查具有挑战性的问题的独特解决方案，否则在孤立地分析每个观点时将很难克服这些问题。主要的研究方向将是：(1)创建一个基准的第一人称和第三人称视频数据集来研究这个新问题；以及开发算法，(2)学习在两个视图之间建立对象和人类的对应关系；(3)学习跨视图的对象-动作关系；以及(4)匿名视觉数据以保护隐私。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

期刊论文数量（9）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Recognizing Actions in Videos from Unseen Viewpoints

DOI：
10.1109/cvpr46437.2021.00411
发表时间：
2021-03
期刊：
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
影响因子：
0
作者：
A. Piergiovanni;M. Ryoo
通讯作者：
A. Piergiovanni;M. Ryoo

Neural Neural Textures Make Sim2Real Consistent

神经网络纹理使 Sim2Real 保持一致

DOI：
发表时间：
2022
期刊：
Conference on Robot Learning (CoRL
影响因子：
0
作者：
Burgert, Ryan;Shang, Jinghuan;Li, Xiang;Ryoo, Michael S.
通讯作者：
Ryoo, Michael S.

Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learning

DOI：
10.1109/iros51168.2021.9636363
发表时间：
2021-08
期刊：
2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
影响因子：
0
作者：
Jinghuan Shang;M. Ryoo
通讯作者：
Jinghuan Shang;M. Ryoo

Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?