RI: Small: Understanding Hand Interaction In The Jumble of Internet Videos

手是人类与世界互动和操纵世界的主要方式。如果智能机器要理解人类的行为，并在人类用手建立的世界中工作，就需要能够理解人类如何使用他们的手。不幸的是，在目前的人工智能(AI)系统中，显示人们使用双手的视频令人惊讶地难以理解。当人们与物体互动时，手可能会暂时隐藏起来，即使它们是可见的，手也可以与无数不同的物体互动，从冰箱把手到咖啡杯再到车库门开启器。这个项目开发的系统可以从大规模的互联网视频数据中学习人类如何使用手。由于手是许多其他研究领域的核心，这个项目有可能促进许多其他学科的研究。例如，机器人研究人员可能会使用这些系统来教机器人如何通过观察与物体互动。同样，研究如何使用人类手的运动学家和机械工程师可以使用这些系统更好地量化手的运动，从而改善人们的生活。该项目旨在通过三个技术方向实现其目标，这三个方向共同促进了理解人类活动和负担(人/物相互作用)的科学。该项目的第一个方向是建立从大规模视频中自动解析手交互数据的系统。这个方向的目标是理解手在与世界的物理互动方面所做的事情，而不是通过命名与名词和动词的互动。为了帮助理解互动的背景，第二个方向旨在建立基于学习的系统，该系统可以根据视频数据中自然出现的部分观察来理解人体姿势。最后，第三个方向通过构建交互图来将这些系统组合在一起，其中手部交互示例是节点，边是由对人体姿势的观察得出的。这一互动网络将使系统能够了解人类如何跨视点和实例从大规模数据中操纵对象，并使计算机视觉能够得到新的应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Hands are the primary way that humans interact with and manipulate the world. Intelligent machines will need to be able to understand how humans use their hands if they are to understand human actions and to work in the world humans have built with their hands. Unfortunately, videos that show people using their hands are surprisingly difficult to understand for current artificial intelligence (AI) systems. Hands may be temporarily hidden as people interact with objects, and even if they are visible, hands can interact with a myriad of different objects ranging from refrigerator handles to coffee mugs to garage door openers. This project develops systems that can enable learning about how humans use their hands from large scale Internet video data. As hands are central to many other areas of study, this project has the potential to empower research in many other disciplines. For instance, robotics researchers may use the systems to teach robots how to interact with objects by observation. Similarly, kinesiologists and mechanical engineers who study how the human hand is used could use the systems to better quantify hand motions and thus improve the lives of people. This project aims to achieve its goal via three technical directions that together advance the science of understanding human activities and affordances (human/object interaction). The first direction of the project will build systems for automatically parsing hand interaction data from large-scale video. The goal of this direction is to understand what the hand is doing in terms of interaction with the world in physical terms as opposed to via naming the interaction with nouns and verbs. To help understand the context of an interaction, the second direction aims to build learning-based systems that can understand human poses from partial observations that occur naturally in video data. Finally, the third direction puts these systems together by building a graph of interaction where hand interaction examples are nodes, and edges are induced by observations of human pose. This web of interactions will enable systems to learn about how humans can manipulate objects from large-scale data across viewpoints and examples and enable new applications of computer vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.