EAGER: Robots that Learn to Communicate with Humans Tthrough Natural Dialog

这项早期的探索性研究拨款探索了开发更友好、更有能力的机器人的可能性，这些机器人学习理解自然人类语言中的命令。开发的实验系统旨在让用户参与自然对话，澄清无法理解的语言指令，并从这种互动中学习，以更有力地解释未来的命令。这一全新的方法假定是为了克服成本更高的以前方法的限制，这些方法需要直接编程或对每个汇编的语言数据进行详细注释，并且仍然经常无法涵盖实际用户交互中出现的问题。最终的探索性原型在与人类用户的真实互动中进行评估，实验测试其通过正常的日常使用，随着时间的推移提高其解释人类指令的准确性和灵活性的能力。这一新颖的方法旨在改善人类与智能多机器人系统的互动，以帮助大型多用途建筑的居民和游客。这项基础研究还支持自然语言处理、人机交互和机器学习等不断增长的领域的计算机科学教育，这些领域对知识人才的需求很大。探索的技术方法是来自三个目前不同领域的学习技术的新颖集成：语义分析、口语对话管理和感知语言基础。语义分析是使用基于句法语言结构的组合语义将自然语言映射到计算机可解释的形式语言的任务。对话管理关注的是控制多轮自然语言交互，以帮助理解和完成任务。感知基础是指将语言中的单词和短语与机器人传感器感知的世界中的对象、属性和关系联系起来。尽管最近在每个领域都取得了重大进展，但以前还没有人探索将它们整合起来，以支持通过自然对话进行人-机器人交流的学习。这项探索性研究采用并集成了使用组合范畴语法的语义解析器学习、使用部分可观察的马尔可夫决策过程的对话管理以及使用视觉和触觉传感器的多模式语言基础的技术，以开发一种用于与机器人交流的新型对话系统，该系统包括德克萨斯大学奥斯汀分校正在开发的创新的建筑范围智能系统。通过使用在线模拟和众包用户对一系列任务进行控制实验，以及用户与由轮式Segway底座和Kinova机器人手臂组成的移动机器人平台的自然交互，对探索性方法进行了评估。

This EArly Grant for Exploratory Research explores the possibility of developing more user-friendly and capable robots that learn to understand commands in natural human language. The experimental system developed aims to engage users in natural conversation, clarifying linguistic instructions that cannot be understood, and learning from this interaction to more robustly interpret future commands. This fundamentally new approach is hypothesized to overcome limitations of more-costly previous approaches that require either direct programming or detailed annotation of per-assembled linguistic data, and still frequently fail to cover issues that arise in real user interactions. The resulting exploratory prototype is evaluated on real interactions with human users, experimentally testing its ability to improve its accuracy and flexibility at interpreting human instructions over time, through normal everyday use. This novel approach aims to improve human interaction with intelligent multi-robot systems that aid the residents and visitors of a large, multi-use building. This fundamental research also supports computer-science education in the growing areas of natural-language processing, human-robot interaction, and machine learning, where there is significant national demand for knowledgeable personnel.The technical approach explored is a novel integration of learning techniques from three currently disparate areas: semantic parsing, spoken dialog management, and perceptual language grounding. Semantic parsing is the task of mapping natural language to a formal computer-interpretable language using compositional semantics based on syntactic linguistic structure. Dialog management concerns controlling multi-turn natural language interaction to aid comprehension and task completion. Perceptual grounding concerns associating words and phrases in language to objects, properties and relations in the world as perceived by the robot's sensors. Although there has been recent significant progress in each of these individual areas, no one has previously explored integrating them to support learning for human-robot communication through natural dialog. This exploratory research adapts and integrates techniques for semantic-parser learning using combinatory categorial grammar, dialog management using Partially Observable Markov Decision Processes, and multi-modal language grounding using both visual and haptic sensors, in order to develop a novel dialog system for communicating with robots that comprise the innovative Building Wide Intelligence system being developed at the University of Texas at Austin. The exploratory methods are evaluated using controlled experiments on a range of tasks using both on-line simulations and crowdsourced users, and natural user interaction with a mobile robot platform consisting of a wheeled Segway base and a Kinova robot arm.