CAREER: 4D Human-Object Interaction Understanding in the Wild

职业：野外 4D 人机交互理解

• 批准号: 2240014
• 负责人: Xiaolong Wang
• 金额: $ 50万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-05-01 至 2028-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2240014&HistoricalAwards=false
• 关键词: CAREER; 4D; Human; Object; Interaction

For many applications in robotics, social science, and medical analysis, it is critical to have computers to understand how humans interact with the physical world. The goal of this project is to build an artificial perception system that goes beyond 2D visual understanding and focuses on parsing the time dynamics and 3D structure of human-object interaction. Thus, to allow real-world applications, the system for understanding 4D (3D + time) human-object interaction is designed to generalize to unconstrained environments of daily life. The developed technologies will enable household robots to predict, and assist humans and doctors to analyze the behaviors of the elderly for providing better care, etc. The education and outreach activities of the project will advance the theoretical courses in computer vision and robotics, and offer practical guidance for undergraduate, graduate, and K-12 students to study in this field.This research develops a 4D human-object interaction understanding system, with a focus on generalization to environments in the wild. Existing approaches are limited to operating within constrained labs or uncluttered scenes. This is mainly due to current computer vision methods heavily relying on detailed 3D/4D annotations, which are usually unavailable for unconstrained environments. To address this limitation, the project provides a framework that will (i) recognize objects from videos in an open-world setting using only text-supervision and self-supervision; (ii) reconstruct and estimate the human and object structures in 3D using minimum 3D supervision and manually defined priors; and (iii) perform 4D interaction and contact relation reasoning between humans and objects using a 4D Interaction Graph. Based on the research studies, the project will develop a 4D Vision Education Platform on course materials focusing on both the generalization aspect of computer vision and the applications in robotic tasks. This platform will be applied in undergraduate and graduate courses, and tutorials for K-12 students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对于机器人学、社会科学和医学分析中的许多应用程序来说，让计算机了解人类如何与物理世界交互是至关重要的。这个项目的目标是建立一个超越2D视觉理解的人工感知系统，专注于解析人与物体交互的时间动态和3D结构。因此，为了允许现实世界的应用，用于理解4D(3D+时间)人-物交互的系统被设计为推广到日常生活的不受限制的环境。开发的技术将使家用机器人能够预测并帮助人类和医生分析老年人的行为，从而提供更好的护理等。该项目的教育和推广活动将推进计算机视觉和机器人学的理论课程，并为本科生、研究生和K-12学生在该领域的学习提供实践指导。本研究开发了一个4D人-物交互理解系统，重点是对野外环境的泛化。现有的方法仅限于在受限的实验室或整洁的场景中操作。这主要是由于当前的计算机视觉方法严重依赖于详细的3D/4D注释，而这些注释通常不适用于不受约束的环境。为了解决这一局限性，该项目提供了一个框架，它将(I)仅使用文本监督和自我监督来识别开放世界环境中的视频对象；(Ii)使用最小3D监督和手动定义的先验在3D中重建和估计人和对象的结构；以及(Iii)使用4D交互图执行4D交互和人与对象之间的接触关系推理。在研究研究的基础上，该项目将开发一个关于课程材料的4D视觉教育平台，重点关注计算机视觉的泛化方面和在机器人任务中的应用。该平台将应用于本科生和研究生课程，以及K-12学生的教程。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。