NRI: Robots that Learn to Communicate through Natural Human Dialog

NRI：通过自然人类对话学习交流的机器人

• 批准号: 1637736
• 负责人: Raymond Mooney
• 金额: $ 93.69万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1637736&HistoricalAwards=false
• 关键词: NRI; Robots; Learn; Communicate; through

Robots are increasingly capable and are on the threshold of becoming a ubiquitous technology. For robots to be truly useful, people must be able to effectively communicate their needs in everyday human language. Although there is a growing body of research on natural-language processing for human-robot interaction, it typically requires some form of explicit supervision provided by an engineering expert and involves unnatural, laborious training to obtain robustness and coverage. This project involves the development of human-robot dialog systems that learn to communicate with users through natural dialog, learning from repeated normal user interactions to become more robust and capable. The project supports the education of students in the areas of natural-language processing, human-robot interaction, and machine learning, where there is significant demand for educated personnel. It is integrated with the university's Freshman Research Initiative, which gets undergraduate students involved in research in their first year.In order to develop human-robot dialog systems that learn to improve their communication skills through normal user interactions, the project integrates and adapts learning techniques from three currently disparate technical areas: semantic parsing, spoken dialog management, and perceptual language grounding. The project adapts and integrates techniques for semantic-parser learning using combinatory categorial grammar (CCG), dialog management using Partially Observable Markov Decision Processes (POMDPs), and multi-modal language grounding using both visual and haptic sensors, in order to develop a dialog system for communicating with robots that comprise the Building Wide Intelligence (BWI) system being developed at the University of Texas at Austin. The research integrates the PI's expertise in semantic parsing and language grounding with the co-PI's expertise in robotics and reinforcement learning, forming a unique interdisciplinary team for developing novel and effective systems for human-robot interaction. The project includes rigorous evaluations using controlled experiments on a range of tasks using both on-line simulations with crowdsourced users, and natural user interaction with a mobile robot platform consisting of a wheeled Segway base and a Kinova robot arm being developed for the BWI system.

机器人的能力越来越强，即将成为一项无处不在的技术。为了让机器人真正有用，人们必须能够用日常的人类语言有效地表达他们的需求。尽管关于人机交互的自然语言处理的研究越来越多，但它通常需要工程专家提供某种形式的明确监督，并涉及非自然的、费力的训练，以获得鲁棒性和覆盖范围。该项目涉及开发人机对话系统，学习通过自然对话与用户交流，从重复的正常用户交互中学习，变得更加强大和有能力。该项目支持学生在自然语言处理、人机交互和机器学习领域的教育，这些领域对受过教育的人员有很大的需求。它与该校的新生研究计划（Freshman Research Initiative）相结合，后者让本科生在第一年就参与研究。为了开发人机对话系统，通过正常的用户交互来学习提高他们的沟通技巧，该项目集成并适应了目前三个不同技术领域的学习技术：语义解析、口语对话管理和感知语言基础。该项目采用并集成了使用组合分类语法（CCG）的语义解析器学习技术，使用部分可观察马尔可夫决策过程（pomdp）的对话管理技术，以及使用视觉和触觉传感器的多模态语言基础技术，以开发一个对话系统，用于与组成建筑智能（BWI）系统的机器人进行通信，该系统由德克萨斯大学奥斯汀分校开发。该研究将PI在语义解析和语言基础方面的专业知识与co-PI在机器人和强化学习方面的专业知识相结合，形成了一个独特的跨学科团队，用于开发新颖有效的人机交互系统。该项目包括对一系列任务进行严格的评估，使用众包用户的在线模拟，以及用户与移动机器人平台的自然交互，该平台由轮式Segway基座和为BWI系统开发的Kinova机器人手臂组成。

项目成果

Dialog as a Vehicle for Lifelong Learning

• 发表时间: 2020-06
• 期刊: ArXiv
• 作者: Aishwarya Padmakumar;R. Mooney

Improving Grounded Natural Language Understanding through Human-Robot Dialog

• DOI: 10.1109/icra.2019.8794287
• 发表时间: 2019-03
• 期刊: 2019 International Conference on Robotics and Automation (ICRA)
• 作者: Jesse Thomason;Aishwarya Padmakumar;Jivko Sinapov;Nick Walker;Yuqian Jiang;Harel Yedidsion;Justin W. Hart;P. Stone;R. Mooney

Improving Black-box Speech Recognition using Semantic Parsing

• 发表时间: 2017-11
• 作者: Rodolfo Corona;Jesse Thomason;R. Mooney

Optimal Use Of Verbal Instructions For Multi-Robot Human Navigation Guidance

多机器人人类导航指导中口头指令的优化使用

• DOI: 10.1007/978-3-030-35888-4_13
• 发表时间: 2019
• 期刊: Proceedings of the Eleventh International Conference on Social Robotics
• 作者: Yedidsion, H.;Deans, J.;Sheehan, C.;Chillara, M.;Hart, J.;Stone, P.;Mooney, R.
• 通讯作者: Mooney, R.

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

• DOI: 10.1613/jair.1.11485
• 发表时间: 2020-02
• 期刊: J. Artif. Intell. Res.
• 作者: Jesse Thomason;Aishwarya Padmakumar;Jivko Sinapov;Nick Walker;Yuqian Jiang;Harel Yedidsion;Justin W. Hart;P. Stone;R. Mooney