RI: Medium: Robots That Learn From Description Through Synthesis and Analysis

许多类型的体力工作在协作中最有效地完成。例如，汽车修理工、电工、外科医生和木匠经常雇用助手或学徒来帮助他们完成工作。这个项目的目标是开发一些基本原理，使智能机器人系统能够自然地与人类伙伴合作完成这些任务。特别地，提出的研究研究了在响应复杂命令时执行动作的问题；例如，“当我把这两件东西放在一起，把孔对齐时，在孔里放一个3英寸的螺丝。”这项研究的结果将在一个能够协助典型组装或烹饪任务的系统中得到证明。该研究探索了基于感知的分类器的发展，这些分类器是根据命令本身的结构根据需要组成的。该分类器由预训练的通用组件构建，然后使用同样源自命令结构的模拟生成的数据进行微调。这带来了几个具有挑战性的技术问题。第一个问题是创建通用的感知组件，这些组件对应于查询的对象、操作、关系和活动的语言单位。第二个问题是将这些单元组合成一个分类器，并根据仿真数据创建按需微调分类器的方法。然后，系统必须能够命令机器人根据命令执行正确的动作。衡量这项研究是否成功的标准是：1)系统能够提供的能力的广度；2)应用于代表性视频数据时所能达到的专用性；3)与人类用户自然交互以执行协作操作任务的能力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Many types of physical work are performed most efficiently in collaboration. For example, garage mechanics, electricians, surgeons, and carpenters often employ an assistant or an apprentice to help them perform their jobs. The goal of the proposed project is to develop the underlying principles that would allow an intelligent robotic system to naturally collaborate with a human partner in such tasks. In particular, the proposed research studies the problem of performing an action in response to a complex command; for example, "When I hold these two pieces together with the holes aligned, place a 3 inch screw through the hole." The results of this research will be demonstrated in a system that is able to assist in typical assembly or cooking tasks.The research explores the development of perception-based classifiers that are composed, on demand, from the structure of the command itself. This classifier is constructed from pre-trained generic components and is then fine-tuned using simulation-generated data also derived from the structure of the command. This poses several challenging technical problems. The first problem is to create general-purpose perceptual components that correspond to the language units of the query's objects, actions, relationships, and activities. The second problem is to compose these units to form a classifier and to create methods for on-demand fine-tuning of the classifier from simulation data. The system must then be able to command a robot to perform the correct action in response to the command. Measures of success of this research will be 1) the breadth of capabilities the system is able to provide; 2) the specificity it is able to achieve when applied to representative video data; and 3) its ability to naturally interact with a human user to perform a collaborative manipulation task.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.