RI: Small: Understanding Hand Interaction In The Jumble of Internet Videos

RI：小：在混乱的互联网视频中理解手部交互

• 批准号: 2006619
• 负责人: David Fouhey
• 金额: $ 43.7万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2006619&HistoricalAwards=false
• 关键词: RI; Small; Understanding; Hand; Interaction

Hands are the primary way that humans interact with and manipulate the world. Intelligent machines will need to be able to understand how humans use their hands if they are to understand human actions and to work in the world humans have built with their hands. Unfortunately, videos that show people using their hands are surprisingly difficult to understand for current artificial intelligence (AI) systems. Hands may be temporarily hidden as people interact with objects, and even if they are visible, hands can interact with a myriad of different objects ranging from refrigerator handles to coffee mugs to garage door openers. This project develops systems that can enable learning about how humans use their hands from large scale Internet video data. As hands are central to many other areas of study, this project has the potential to empower research in many other disciplines. For instance, robotics researchers may use the systems to teach robots how to interact with objects by observation. Similarly, kinesiologists and mechanical engineers who study how the human hand is used could use the systems to better quantify hand motions and thus improve the lives of people. This project aims to achieve its goal via three technical directions that together advance the science of understanding human activities and affordances (human/object interaction). The first direction of the project will build systems for automatically parsing hand interaction data from large-scale video. The goal of this direction is to understand what the hand is doing in terms of interaction with the world in physical terms as opposed to via naming the interaction with nouns and verbs. To help understand the context of an interaction, the second direction aims to build learning-based systems that can understand human poses from partial observations that occur naturally in video data. Finally, the third direction puts these systems together by building a graph of interaction where hand interaction examples are nodes, and edges are induced by observations of human pose. This web of interactions will enable systems to learn about how humans can manipulate objects from large-scale data across viewpoints and examples and enable new applications of computer vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

手是人类与世界互动和操纵世界的主要方式。如果智能机器要理解人类的行为，并在人类用双手建造的世界中工作，它们就需要能够理解人类如何使用双手。不幸的是，对于当前的人工智能（AI）系统来说，显示人们使用双手的视频令人惊讶地难以理解。当人们与物体互动时，手可能会暂时隐藏起来，即使它们是可见的，手也可以与无数不同的物体互动，从冰箱把手到咖啡杯再到车库开门器。该项目开发的系统可以从大规模的互联网视频数据中了解人类如何使用双手。由于手是研究的许多其他领域的核心，这个项目有可能在许多其他学科的研究授权。例如，机器人研究人员可以使用这些系统来教机器人如何通过观察与物体进行交互。同样，研究人手使用方式的运动学家和机械工程师可以使用该系统更好地量化手部运动，从而改善人们的生活。该项目旨在通过三个技术方向实现其目标，共同推进理解人类活动和启示（人/物交互）的科学。该项目的第一个方向将建立从大规模视频中自动解析手部交互数据的系统。这个方向的目标是理解手在与物理世界的交互方面正在做什么，而不是通过用名词和动词命名交互。为了帮助理解交互的上下文，第二个方向旨在构建基于学习的系统，该系统可以从视频数据中自然发生的部分观察中理解人类姿势。最后，第三个方向通过构建交互图将这些系统放在一起，其中手交互示例是节点，边由对人类姿势的观察引起。这个交互网络将使系统能够了解人类如何从大规模数据中跨视角和示例操纵对象，并实现计算机视觉的新应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

MOVES: Manipulated Objects in Video Enable Segmentation

• DOI: 10.1109/cvpr52729.2023.00613
• 发表时间: 2023-06
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Richard E. L. Higgins;D. Fouhey

COHESIV: Contrastive Object and Hand Embeddings for Segmentation In Video

COHESIV：用于视频分割的对比对象和手嵌入

• 发表时间: 2021
• 期刊: Advances in neural information processing systems
• 作者: Shan, Dandan;Higgins, Richard E.L.;Fouhey, David F.
• 通讯作者: Fouhey, David F.

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

EPIC-KITCHENS VISOR 基准：视频分段和对象关系

• 发表时间: 2022
• 期刊: NeurIPS
• 作者: Darkhalil, Ahmad;Shan, Dandan;Zhu, Bin;Ma, Jian;Kar, Amlan;Higgins, Richard;Fidler, Sanja;Fouhey, David;Damen, Dima
• 通讯作者: Damen, Dima