RI:Medium:Collaborative Research: Object-Centric Inference of Actionable Information from Visual Data

该项目将创建新颖的算法和学习架构，适合于理解如何在有目的的对象操作环境中计划和执行动作。这种理解对于在非结构化环境中操作的自主代理是必不可少的，并且在执行各种物理任务期间为人类提供自动化帮助也很有价值。该项目将通过计算“想象”具有类似人类操作能力的参与者可以在该环境中做出的变化，并生成可以完成所需操作的计划。这些工具有助于创造智能环境，例如，观察老年人的感知系统可以推断出该人试图完成的任务，并提供建议/帮助。它们还允许创建针对特定环境定制的自动化教学视频，可用于有效培训非技术工人。该项目将为各种各样的学生提供指导和研究机会，包括在计算机科学领域代表性不足的群体的成员。这项研究将研究由物体形成的环境，其中一些是可以操纵的，而另一些则定义了要避免的障碍物或要使用的支撑表面。操作对象通常意味着与对象的一小部分进行交互，这些部分被称为活动区域：手柄、按钮、杠杆、可抓取或可推的区域等。一个深层次的挑战是开发工具来大规模地识别和分类物体上的这些活动位点，并根据动态2D/3D图像编纂它们参与的交互类型，建立基本动作的词汇表。这需要新颖的机器学习方法和深度架构来处理大规模动态视觉和几何数据。它还需要在更抽象的层面上描述操作的特征，以便它们可以被各种效应器（机器人或人类）用于不同的物体几何形状和物理特征。随着在线对象模型存储库（如ShapeNet）接收到更多的可视化数据，进一步的挑战是可操作信息的积累和更新。该方法的最后但关键的一步将是开发工具，用于将这些动作知识传输到与捕获设置相似但不相同的新设置，使用各种数学工具，包括功能图。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will create novel algorithms and learning architectures suitable for understanding how to plan and execute actions in an environment for purposeful object manipulation. Such understanding is indispensable for autonomous agents operating in unstructured environments, and it is also valuable in providing automated assistance to humans during the execution of various physical tasks. The project will computationally "imagine" changes that actors with human-like manipulation capabilities can make on that environment and generate plans that can accomplish the desired manipulations. Such tools facilitate the creation of smart environments, where for example a perception system watching an elderly person can infer the task the person is trying to accomplish and offer advice/assistance. They also allow the creation of automated instructional videos customized to a particular environment that can be used for efficient training of unskilled workers. The project will provide mentoring and research opportunities for a diverse set of students, including members of groups typically under-represented in computer science.This research will study environments formed by objects, some of which can be manipulated, while others define obstacles to be avoided or support surfaces to be used. Manipulating an object typically means interacting with small parts of the object, referred to as its active sites: handles, buttons, levers, graspable or pushable regions, etc. A deep challenge is to develop tools for identifying and classifying these active sites on objects at large scale, and to codify the types of interactions they partake of based on dynamic 2D/3D imagery, building a vocabulary of elementary actions. This requires novel machine learning methods and deep architectures for processing large-scale dynamic visual and geometric data. It also requires characterizing manipulations at a more abstract level so that they can be used by a variety of effectors, robotic or human, on different object geometries and physical characteristics. A further challenge is the accumulation and update of actionable information as more visual data is received in online object model repositories, such as ShapeNet. A final but key step of the approach will be the development of tools for transporting such action knowledge to new settings that are similar but not identical to the capture settings, using a variety of mathematical tools including functional maps.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.