RI:Small:Collaborative Research: Understanding Human-Object Interactions from First-person and Third-person Videos

RI:Small：协作研究：从第一人称和第三人称视频中理解人与物体的交互

• 批准号: 2204808
• 负责人: Yong Jae Lee
• 金额: $ 15万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-11-01 至 2022-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2204808&HistoricalAwards=false
• 关键词: RI; Small; Collaborative; Research; Understanding

Ubiquitous cameras, together with ever increasing computing resources, are dramatically changing the nature of visual data and their analysis. Cities are adopting networked camera systems for policing and intelligent resource allocation, and individuals are recording their lives using wearable devices. For these camera systems to become truly smart and useful for people, it is crucial that they understand interesting objects in the scene and detect ongoing activities/events, while jointly considering continuous 24/7 videos from multiple sources. Such object-level and activity-level awareness in hospitals, elderly homes, and public places would provide assistive and quality-of-life technology for disabled and elderly people, provide intelligent surveillance systems to prevent crimes, and allow smart usage of environmental resources. This project will investigate novel computer vision algorithms that combine 1st-person videos (from wearable cameras) and 3rd-person videos (from static environmental cameras) for joint recognition of humans, objects, and their interactions. The key idea is to combine the two views' complementary and unique advantages for joint visual scene understanding. To this end, it will create a new dataset, and develop new algorithms that learn to recognize objects jointly across the views, learn human-object and human-human relationships through the two views, and anonymize the videos to preserve users' privacies. The project will provide new algorithms that have the potential to benefit applications in smart environments, security, and quality-of-life assistive technologies. The project will also perform complementary educational and outreach activities that engage students in research and STEM.This project will develop novel algorithms that learn from joint 1st-person videos (from wearable cameras) and 3rd-person videos (from static environmental cameras) for joint recognition of humans, objects, and their interactions. The 1st-person view is ideal for object recognition, while the 3rd-person view is ideal for human activity recognition. Thus, this project will investigate unique solutions to challenging problems that would otherwise be difficult to overcome when analyzing each viewpoint in isolation. The main research directions will be: (1) creating a benchmark 1st-person and 3rd-person video dataset to investigate this new problem; and developing algorithms that (2) learn to establish object and human correspondences between the two views; (3) learn object-action relationships across the views; and (4) anonymize the visual data for privacy-preserving visual recognition.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

无处不在的摄像机，以及不断增加的计算资源，正在极大地改变视觉数据的性质及其分析。城市正在采用网络相机系统进行警务和智能资源分配，个人正在使用可穿戴设备记录生活。对于这些相机系统对人们来说真正聪明和有用，至关重要的是，他们了解场景中有趣的对象并检测正在进行的活动/事件，同时共同考虑来自多个来源的连续24/7视频。这种对象级别和活动水平的意识在医院，老年家庭和公共场所将为残疾人和老年人提供辅助和生活质量的技术，提供智能监视系统以防止犯罪，并允许对环境资源进行明智的使用。 该项目将研究结合第一人称视频（来自可穿戴摄像机）和第三人称视频（来自静态环境摄像机）的新型计算机视觉算法，以共同识别人类，对象及其相互作用。关键的想法是将两种视图的互补和独特优势结合在一起，以理解联合视觉场景。为此，它将创建一个新的数据集，并开发新的算法，这些算法学会通过两种观点来识别对象，学习人类对象和人类关系，并将视频匿名化以保留用户的私有性。该项目将提供新的算法，这些算法有可能使应用程序在智能环境，安全性和生活质量辅助技术中受益。该项目还将进行互补的教育和外展活动，使学生参与研究和STEM。该项目将开发新的算法，这些算法从联合的第一人称视频（来自可穿戴式相机）和第三人称视频（来自静态环境摄像机）（来自静态环境摄像机）学习，以共同识别人类，对象及其相互作用。第一人称视图是对象识别的理想选择，而第三人称视图是人类活动识别的理想选择。因此，该项目将调查针对挑战性问题的独特解决方案，这些解决方案在孤立分析每个观点时将难以克服。主要的研究指示将是：（1）创建基准的第一人称和第三人称视频数据集来研究这个新问题；并开发（2）学会在两种观点之间建立对象和人类对应的算法； （3）学习跨视图的对象关系关系； （4）匿名为保护隐私视觉识别的视觉数据。该奖项反映了NSF的法定任务，并且使用基金会的知识分子优点和更广泛的影响评估标准，被认为值得通过评估来获得支持。

项目成果

Seeing the Unseen: Predicting the First-Person Camera Wearer’s Location and Pose in Third-Person Scenes

• DOI: 10.1109/iccvw54120.2021.00384
• 发表时间: 2021-10
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
• 作者: Yangming Wen;Krishna Kumar Singh;Markham H. Anderson;Wei-Pang Jan;Yong Jae Lee

The Two Dimensions of Worst-case Training and Their Integrated Effect for Out-of-domain Generalization

• DOI: 10.1109/cvpr52688.2022.00941
• 发表时间: 2022-04
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Zeyi Huang;Haohan Wang;Dong Huang;Yong Jae Lee;Eric P. Xing

Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features

• 发表时间: 2021-11
• 作者: Haohan Wang;Zeyi Huang;Hanlin Zhang;Eric P. Xing

Equine Pain Behavior Classification via Self-Supervised Disentangled Pose Representation

• DOI: 10.1109/wacv51458.2022.00023
• 发表时间: 2021-08
• 期刊: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: M. Rashid;S. Broomé;K. Ask;Elin Hernlund;P. Andersen;H. Kjellström;Yong Jae Lee

Delving Deeper into Anti-Aliasing in ConvNets

• DOI: 10.1007/s11263-022-01672-y
• 发表时间: 2020-08
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Xueyan Zou;Fanyi Xiao;Zhiding Yu;Yong Jae Lee