Collaborative Research: Inverse Task Planning from Few-Shot Vision Language Demonstrations
协作研究:基于少镜头视觉语言演示的逆向任务规划
基本信息
- 批准号:2327974
- 负责人:
- 金额:$ 45万
- 依托单位:
- 依托单位国家:美国
- 项目类别:Standard Grant
- 财政年份:2024
- 资助国家:美国
- 起止时间:2024-01-01 至 2026-12-31
- 项目状态:未结题
- 来源:
- 关键词:
项目摘要
This project advances national prosperity and welfare by taking a step towards making collaborative robots more accessible across various settings, such as homes, factories, and logistics operations. Today, robots can only be programmed by engineers, limiting the tasks they can do to a narrow set of design choices made by these engineers. For robots to be widely adopted in households, they need to be versatile enough to tackle a much broader range of tasks. Many of these tasks, such as preparing meals or organizing the home, require personalization to the individual user's needs and preferences. This project aims to enable a robot to learn such personalized tasks through natural interactions with the user who may provide visual demonstrations combined with language narration to describe the task. Neither vision nor language alone is perfect, but together, they capture the task that the user wants to convey. This research leverages the broad impact Large Language Model (LLM) interfaces, such as ChatGPT, has had on engaging with everyday users and brings that into the physical realm with robots. It addresses fundamental challenges like summarizing vision-language demonstrations, verifying automatically generated plans and efficiently solving tasks that require many steps to achieve a goal. If successful, the project could transform many consumer robotics applications, enabling robots to be more usable, personalized and aligned with user values.The primary objective of this project is to develop a framework for learning complex, long-horizon tasks from few-shot vision-language demonstrations. While existing approaches in Inverse Reinforcement Learning (IRL) enable learning simple, short-horizon skills from demonstrations, scaling these approaches to longer horizons with fewer demonstrations poses fundamental statistical and computational challenges. To address these challenges, a novel framework called Inverse Task Planning (ITP) that combines the generalization power of Large Language Models (LLMs) with performance guarantees of IRL to both efficiently and verifiably learn tasks will be used. This approach is uniquely different from existing work in LLM and task planning as it creates a closed-loop system to align LLM outputs with human demonstrations. Concretely, the plan is to: (1) parse vision-language demonstrations as robot state-action trajectories using visual question answering (2) learn language-based reward summaries from long-horizon state-action trajectories, and (3) optimize rewards by generating high-level task-code in a verifiable, closed-loop fashion. This research has broad implications for creating new interfaces that allow everyday users to program robots, developing courses on generative models and robotics, and providing immersive and engaging programming activities for K-12 students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
该项目通过使协作机器人在家庭、工厂和物流运营等各种环境中更容易使用,从而促进了国家的繁荣和福利。今天,机器人只能由工程师编程,将它们能做的任务限制在这些工程师设计的一组狭窄的选择中。为了让机器人在家庭中被广泛采用,它们需要足够多的功能来处理更广泛的任务。许多这样的任务,如准备饭菜或整理家庭,都需要个性化,以满足个人用户的需求和偏好。本项目旨在通过与用户的自然交互,使机器人能够学习这种个性化的任务,用户可以提供视觉演示并结合语言叙述来描述任务。视觉和语言本身都不是完美的,但它们结合在一起,就能捕捉到用户想要传达的任务。这项研究利用了诸如ChatGPT等大型语言模型(LLM)接口在与日常用户互动方面的广泛影响,并将其带入机器人的物理领域。它解决了一些基本的挑战,比如总结视觉语言演示,验证自动生成的计划,以及有效地解决需要许多步骤才能实现目标的任务。如果成功,该项目可能会改变许多消费者机器人应用,使机器人更可用,更个性化,更符合用户价值。该项目的主要目标是开发一个框架,用于从少量视觉语言演示中学习复杂的、长期的任务。虽然逆强化学习(IRL)中的现有方法可以从演示中学习简单、短期的技能,但将这些方法扩展到更长的范围、更少的演示会带来基本的统计和计算挑战。为了应对这些挑战,将使用一种称为逆任务规划(ITP)的新框架,该框架将大型语言模型(llm)的泛化能力与IRL的性能保证相结合,以有效且可验证地学习任务。这种方法与LLM和任务规划中的现有工作独特不同,因为它创建了一个闭环系统,以使LLM输出与人类演示保持一致。具体而言,该计划是:(1)使用视觉问答将视觉语言演示解析为机器人状态-动作轨迹;(2)从长期状态-动作轨迹中学习基于语言的奖励摘要;(3)通过以可验证的闭环方式生成高级任务代码来优化奖励。这项研究对于创建新的界面具有广泛的意义,这些界面允许日常用户对机器人进行编程,开发生成模型和机器人课程,并为K-12学生提供沉浸式和引人入胜的编程活动。该奖项反映了美国国家科学基金会的法定使命,并通过使用基金会的知识价值和更广泛的影响审查标准进行评估,被认为值得支持。
项目成果
期刊论文数量(0)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
数据更新时间:{{ journalArticles.updateTime }}
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
数据更新时间:{{ journalArticles.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ monograph.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ sciAawards.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ conferencePapers.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ patent.updateTime }}
Jeannette Bohg其他文献
COAST: Constraints and Streams for Task and Motion Planning
COAST:任务和运动规划的约束和流
- DOI:
10.48550/arxiv.2405.08572 - 发表时间:
2024 - 期刊:
- 影响因子:0
- 作者:
Brandon Vu;Toki Migimatsu;Jeannette Bohg - 通讯作者:
Jeannette Bohg
SpringGrasp: Synthesizing Compliant, Dexterous Grasps under Shape Uncertainty
SpringGrasp:在形状不确定的情况下综合顺应、灵巧的抓取
- DOI:
- 发表时间:
2024 - 期刊:
- 影响因子:0
- 作者:
Sirui Chen;Jeannette Bohg;C. K. Liu - 通讯作者:
C. K. Liu
Jeannette Bohg的其他文献
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
相似国自然基金
Research on Quantum Field Theory without a Lagrangian Description
- 批准号:24ZR1403900
- 批准年份:2024
- 资助金额:0.0 万元
- 项目类别:省市级项目
Cell Research
- 批准号:31224802
- 批准年份:2012
- 资助金额:24.0 万元
- 项目类别:专项基金项目
Cell Research
- 批准号:31024804
- 批准年份:2010
- 资助金额:24.0 万元
- 项目类别:专项基金项目
Cell Research (细胞研究)
- 批准号:30824808
- 批准年份:2008
- 资助金额:24.0 万元
- 项目类别:专项基金项目
Research on the Rapid Growth Mechanism of KDP Crystal
- 批准号:10774081
- 批准年份:2007
- 资助金额:45.0 万元
- 项目类别:面上项目
相似海外基金
NSFGEO-NERC: Collaborative Research: Exploring AMOC controls on the North Atlantic carbon sink using novel inverse and data-constrained models (EXPLANATIONS)
NSFGEO-NERC:合作研究:使用新颖的逆向模型和数据约束模型探索 AMOC 对北大西洋碳汇的控制(解释)
- 批准号:
2347992 - 财政年份:2024
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
NSFGEO-NERC: Collaborative Research: Exploring AMOC controls on the North Atlantic carbon sink using novel inverse and data-constrained models (EXPLANATIONS)
NSFGEO-NERC:合作研究:使用新颖的逆向模型和数据约束模型探索 AMOC 对北大西洋碳汇的控制(解释)
- 批准号:
2347991 - 财政年份:2024
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
Collaborative Research: Inverse Task Planning from Few-Shot Vision Language Demonstrations
协作研究:基于少镜头视觉语言演示的逆向任务规划
- 批准号:
2327973 - 财政年份:2024
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
CDS&E/Collaborative Research: Data-Driven Inverse Design of Additively Manufacturable Aperiodic Architected Cellular Materials
CDS
- 批准号:
2245298 - 财政年份:2023
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
CDS&E/Collaborative Research: Data-Driven Inverse Design of Additively Manufacturable Aperiodic Architected Cellular Materials
CDS
- 批准号:
2245299 - 财政年份:2023
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
Collaborative Research: Breaking the 1D barrier in radiative transfer: Fast, low-memory numerical methods for enabling inverse problems and machine learning emulators
合作研究:打破辐射传输中的一维障碍:用于实现逆问题和机器学习模拟器的快速、低内存数值方法
- 批准号:
2324369 - 财政年份:2023
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
Collaborative Research: Breaking the 1D barrier in radiative transfer: Fast, low-memory numerical methods for enabling inverse problems and machine learning emulators
合作研究:打破辐射传输中的一维障碍:用于实现逆问题和机器学习模拟器的快速、低内存数值方法
- 批准号:
2324368 - 财政年份:2023
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
Collaborative Research: Algorithms for Learning Regularizations of Inverse Problems with High Data Heterogeneity
合作研究:高数据异质性逆问题的学习正则化算法
- 批准号:
2152960 - 财政年份:2022
- 资助金额:
$ 45万 - 项目类别:
Continuing Grant
DMREF/Collaborative Research: Inverse Design of Architected Materials with Prescribed Behaviors via Graph Based Networks and Additive Manufacturing
DMREF/协作研究:通过基于图形的网络和增材制造对具有规定行为的建筑材料进行逆向设计
- 批准号:
2119643 - 财政年份:2022
- 资助金额:
$ 45万 - 项目类别:
Standard Grant
DMREF/Collaborative Research: Inverse Design of Architected Materials with Prescribed Behaviors via Graph Based Networks and Additive Manufacturing
DMREF/协作研究:通过基于图形的网络和增材制造对具有规定行为的建筑材料进行逆向设计
- 批准号:
2119545 - 财政年份:2022
- 资助金额:
$ 45万 - 项目类别:
Standard Grant