RI:Medium:Collaborative Research: Object-Centric Inference of Actionable Information from Visual Data

该项目将创建新的算法和学习架构，适合于理解如何在有目的的对象操作环境中计划和执行操作。这种理解对于在非结构化环境中运行的自主代理是必不可少的，并且在执行各种物理任务期间为人类提供自动化帮助也是有价值的。该项目将通过计算“想象”具有类似人类操作能力的参与者可以在该环境中进行的更改，并生成可以完成所需操作的计划。这些工具有助于创建智能环境，例如，观察老年人的感知系统可以推断出该人试图完成的任务并提供建议/帮助。它们还允许创建针对特定环境定制的自动化教学视频，可用于有效培训非熟练工人。该项目将为不同类型的学生提供指导和研究机会，包括计算机科学中通常代表性不足的群体成员。该研究将研究由对象形成的环境，其中一些可以操纵，而另一些则定义要避免的障碍或要使用的支持表面。操纵对象通常意味着与对象的小部分进行交互，这些小部分被称为其活动站点：手柄、按钮、杠杆、可抓握或可推动区域等。一个深层次的挑战是开发用于识别和分类大规模对象上的这些活动站点的工具，并根据动态2D/3D图像对它们参与的交互类型进行编码，建立基本动作的词汇表。这需要新的机器学习方法和深度架构来处理大规模动态视觉和几何数据。它还需要在更抽象的层次上表征操作，以便它们可以被各种效应器（机器人或人类）用于不同的对象几何形状和物理特性。另一个挑战是随着在线对象模型存储库（如ShapeNet）接收到更多的可视化数据，可操作信息的积累和更新。该方法的最后但关键的一步将是开发工具，用于将此类行动知识传输到与捕获设置相似但不相同的新设置，使用各种数学工具，包括功能图。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

This project will create novel algorithms and learning architectures suitable for understanding how to plan and execute actions in an environment for purposeful object manipulation. Such understanding is indispensable for autonomous agents operating in unstructured environments, and it is also valuable in providing automated assistance to humans during the execution of various physical tasks. The project will computationally "imagine" changes that actors with human-like manipulation capabilities can make on that environment and generate plans that can accomplish the desired manipulations. Such tools facilitate the creation of smart environments, where for example a perception system watching an elderly person can infer the task the person is trying to accomplish and offer advice/assistance. They also allow the creation of automated instructional videos customized to a particular environment that can be used for efficient training of unskilled workers. The project will provide mentoring and research opportunities for a diverse set of students, including members of groups typically under-represented in computer science.This research will study environments formed by objects, some of which can be manipulated, while others define obstacles to be avoided or support surfaces to be used. Manipulating an object typically means interacting with small parts of the object, referred to as its active sites: handles, buttons, levers, graspable or pushable regions, etc. A deep challenge is to develop tools for identifying and classifying these active sites on objects at large scale, and to codify the types of interactions they partake of based on dynamic 2D/3D imagery, building a vocabulary of elementary actions. This requires novel machine learning methods and deep architectures for processing large-scale dynamic visual and geometric data. It also requires characterizing manipulations at a more abstract level so that they can be used by a variety of effectors, robotic or human, on different object geometries and physical characteristics. A further challenge is the accumulation and update of actionable information as more visual data is received in online object model repositories, such as ShapeNet. A final but key step of the approach will be the development of tools for transporting such action knowledge to new settings that are similar but not identical to the capture settings, using a variety of mathematical tools including functional maps.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.