RI: Medium: Robots That Learn From Description Through Synthesis and Analysis

许多类型的体力工作在协作中效率最高。例如，汽车修理工、电工、外科医生和木匠经常雇用助手或学徒来帮助他们完成工作。拟议项目的目标是开发基本原则，使智能机器人系统能够在此类任务中自然地与人类合作。 特别是，拟议的研究研究的问题，执行一个行动，以响应一个复杂的命令;例如，“当我举行这两件在一起的孔对齐，放置一个3英寸的螺丝通过孔。“这项研究的结果将在一个能够帮助典型的组装或烹饪任务的系统中得到证明。这项研究探索了基于感知的分类器的发展，这些分类器是根据命令本身的结构按需组成的。该分类器由预先训练的通用组件构建，然后使用也来自命令结构的模拟生成的数据进行微调。 这提出了几个具有挑战性的技术问题。第一个问题是创建与查询的对象、动作、关系和活动的语言单元相对应的通用感知组件。第二个问题是组合这些单元以形成分类器，并创建根据模拟数据对分类器进行按需微调的方法。 然后，系统必须能够命令机器人响应命令执行正确的动作。衡量这项研究成功的标准是：1）系统能够提供的能力的广度; 2）应用于代表性视频数据时能够实现的特异性;和3）该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识产权进行评估来支持。优点和更广泛的影响审查标准。

Many types of physical work are performed most efficiently in collaboration. For example, garage mechanics, electricians, surgeons, and carpenters often employ an assistant or an apprentice to help them perform their jobs. The goal of the proposed project is to develop the underlying principles that would allow an intelligent robotic system to naturally collaborate with a human partner in such tasks. In particular, the proposed research studies the problem of performing an action in response to a complex command; for example, "When I hold these two pieces together with the holes aligned, place a 3 inch screw through the hole." The results of this research will be demonstrated in a system that is able to assist in typical assembly or cooking tasks.The research explores the development of perception-based classifiers that are composed, on demand, from the structure of the command itself. This classifier is constructed from pre-trained generic components and is then fine-tuned using simulation-generated data also derived from the structure of the command. This poses several challenging technical problems. The first problem is to create general-purpose perceptual components that correspond to the language units of the query's objects, actions, relationships, and activities. The second problem is to compose these units to form a classifier and to create methods for on-demand fine-tuning of the classifier from simulation data. The system must then be able to command a robot to perform the correct action in response to the command. Measures of success of this research will be 1) the breadth of capabilities the system is able to provide; 2) the specificity it is able to achieve when applied to representative video data; and 3) its ability to naturally interact with a human user to perform a collaborative manipulation task.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.