RI:Medium:Collaborative Research: Object-Centric Inference of Actionable Information from Visual Data

这个项目将创建新的算法和学习架构，适合于理解如何在有目的的对象操作环境中计划和执行动作。这种理解对于在非结构化环境中操作的自主代理是必不可少的，对于在执行各种物理任务期间为人类提供自动化帮助也是有价值的。该项目将在计算上“想象”具有类似人类操纵能力的参与者可以对该环境进行的改变，并生成能够完成所需操纵的计划。这类工具有助于创建智能环境，例如，监视老年人的感知系统可以推断该人试图完成的任务，并提供建议/帮助。它们还允许创建针对特定环境定制的自动化教学视频，可用于对非熟练工人进行高效培训。该项目将为不同的学生提供指导和研究机会，包括计算机科学中通常代表不足的群体的成员。这项研究将研究由物体组成的环境，其中一些可以操纵，而另一些则定义要避免的障碍或支持使用的表面。操纵对象通常意味着与对象的小部分交互，称为其活动位置：手柄、按钮、杠杆、可抓取或可推的区域等。一个深层次的挑战是开发工具来识别和分类大规模对象上的这些活动位置，并基于动态2D/3D图像对它们参与的交互类型进行编码，从而建立基本动作的词汇。这就需要新的机器学习方法和深层体系结构来处理大规模的动态视觉和几何数据。它还需要在更抽象的水平上表征操作，以便它们可以被各种效应器使用，无论是机器人还是人类，对不同的对象几何形状和物理特征使用它们。另一个挑战是随着在线对象模型存储库(如ShapeNet)中接收到更多的可视数据，可操作信息的积累和更新。该方法的最后但也是关键的一步是开发工具，使用包括功能地图在内的各种数学工具，将此类行动知识传输到与捕获环境相似但不相同的新环境中。该奖项反映了NSF的法定使命，并已通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will create novel algorithms and learning architectures suitable for understanding how to plan and execute actions in an environment for purposeful object manipulation. Such understanding is indispensable for autonomous agents operating in unstructured environments, and it is also valuable in providing automated assistance to humans during the execution of various physical tasks. The project will computationally "imagine" changes that actors with human-like manipulation capabilities can make on that environment and generate plans that can accomplish the desired manipulations. Such tools facilitate the creation of smart environments, where for example a perception system watching an elderly person can infer the task the person is trying to accomplish and offer advice/assistance. They also allow the creation of automated instructional videos customized to a particular environment that can be used for efficient training of unskilled workers. The project will provide mentoring and research opportunities for a diverse set of students, including members of groups typically under-represented in computer science.This research will study environments formed by objects, some of which can be manipulated, while others define obstacles to be avoided or support surfaces to be used. Manipulating an object typically means interacting with small parts of the object, referred to as its active sites: handles, buttons, levers, graspable or pushable regions, etc. A deep challenge is to develop tools for identifying and classifying these active sites on objects at large scale, and to codify the types of interactions they partake of based on dynamic 2D/3D imagery, building a vocabulary of elementary actions. This requires novel machine learning methods and deep architectures for processing large-scale dynamic visual and geometric data. It also requires characterizing manipulations at a more abstract level so that they can be used by a variety of effectors, robotic or human, on different object geometries and physical characteristics. A further challenge is the accumulation and update of actionable information as more visual data is received in online object model repositories, such as ShapeNet. A final but key step of the approach will be the development of tools for transporting such action knowledge to new settings that are similar but not identical to the capture settings, using a variety of mathematical tools including functional maps.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.