RI: Medium: Robots That Learn From Description Through Synthesis and Analysis

许多类型的体力工作在协作中执行效率最高。例如，车库机械师、电工、外科医生和木匠经常雇佣一名助手或学徒来帮助他们完成自己的工作。拟议项目的目标是开发基本原则，使智能机器人系统能够在此类任务中自然地与人类合作。特别是，这项拟议的研究研究了对复杂命令做出反应时执行动作的问题；例如，当我将这两个部件固定在一起，并将孔对齐时，在孔中插入一颗3英寸的螺丝。这项研究的结果将在一个能够帮助完成典型的装配或烹饪任务的系统中进行演示。该研究探索了基于感知的分类器的发展，这些分类器是根据命令本身的结构按需组成的。该分类器由预先训练的通用组件构建，然后使用模拟生成的数据进行微调，该数据也来自命令的结构。这带来了几个具有挑战性的技术问题。第一个问题是创建与查询的对象、动作、关系和活动的语言单元相对应的通用感知组件。第二个问题是组合这些单元以形成分类器，并根据模拟数据创建按需微调分类器的方法。然后，系统必须能够命令机器人响应命令执行正确的动作。这项研究的成功标准将是1)系统能够提供的能力的广度；2)当应用于代表性视频数据时它能够实现的专用性；3)它与人类用户自然交互以执行协作操纵任务的能力。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Many types of physical work are performed most efficiently in collaboration. For example, garage mechanics, electricians, surgeons, and carpenters often employ an assistant or an apprentice to help them perform their jobs. The goal of the proposed project is to develop the underlying principles that would allow an intelligent robotic system to naturally collaborate with a human partner in such tasks. In particular, the proposed research studies the problem of performing an action in response to a complex command; for example, "When I hold these two pieces together with the holes aligned, place a 3 inch screw through the hole." The results of this research will be demonstrated in a system that is able to assist in typical assembly or cooking tasks.The research explores the development of perception-based classifiers that are composed, on demand, from the structure of the command itself. This classifier is constructed from pre-trained generic components and is then fine-tuned using simulation-generated data also derived from the structure of the command. This poses several challenging technical problems. The first problem is to create general-purpose perceptual components that correspond to the language units of the query's objects, actions, relationships, and activities. The second problem is to compose these units to form a classifier and to create methods for on-demand fine-tuning of the classifier from simulation data. The system must then be able to command a robot to perform the correct action in response to the command. Measures of success of this research will be 1) the breadth of capabilities the system is able to provide; 2) the specificity it is able to achieve when applied to representative video data; and 3) its ability to naturally interact with a human user to perform a collaborative manipulation task.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.