EAGER: Robots that Learn to Communicate with Humans Tthrough Natural Dialog

这项探索性研究的早期拨款探索了开发更友好、更有能力的机器人的可能性，这些机器人可以学习理解人类自然语言的命令。开发的实验系统旨在让用户参与自然对话，澄清无法理解的语言指令，并从这种交互中学习，以更稳健地解释未来的命令。这种基本的新方法被假设为克服以前的方法的局限性，这些方法要么需要直接编程，要么需要对每组装的语言数据进行详细注释，并且仍然经常无法涵盖实际用户交互中出现的问题。由此产生的探索性原型将在与人类用户的真实交互中进行评估，通过日常使用，通过实验测试其随着时间的推移提高解释人类指令的准确性和灵活性的能力。这种新颖的方法旨在改善人类与智能多机器人系统的互动，帮助大型多用途建筑的居民和游客。这项基础研究还支持自然语言处理、人机交互和机器学习等不断发展的领域的计算机科学教育，这些领域对知识渊博的人才有重大的国家需求。所探索的技术方法是一种新的集成学习技术，来自三个目前不同的领域：语义解析，口语对话管理和感知语言基础。语义解析是利用基于句法语言结构的组合语义将自然语言映射为正式的计算机可解释语言的任务。对话管理涉及控制多回合自然语言交互，以帮助理解和完成任务。感知基础涉及将语言中的单词和短语与机器人传感器感知到的世界中的物体、属性和关系联系起来。尽管最近在这些单独的领域都取得了重大进展，但以前没有人探索过将它们整合到通过自然对话进行人机交流的学习中。这项探索性研究采用并集成了使用组合分类语法的语义解析器学习技术，使用部分可观察马尔可夫决策过程的对话管理技术，以及使用视觉和触觉传感器的多模态语言基础技术，以开发一种用于与机器人通信的新型对话系统，该对话系统包括德克萨斯大学奥斯汀分校正在开发的创新建筑智能系统。通过在线模拟和众包用户对一系列任务进行控制实验，以及用户与由轮式Segway基座和Kinova机器人手臂组成的移动机器人平台的自然交互，对探索性方法进行了评估。

This EArly Grant for Exploratory Research explores the possibility of developing more user-friendly and capable robots that learn to understand commands in natural human language. The experimental system developed aims to engage users in natural conversation, clarifying linguistic instructions that cannot be understood, and learning from this interaction to more robustly interpret future commands. This fundamentally new approach is hypothesized to overcome limitations of more-costly previous approaches that require either direct programming or detailed annotation of per-assembled linguistic data, and still frequently fail to cover issues that arise in real user interactions. The resulting exploratory prototype is evaluated on real interactions with human users, experimentally testing its ability to improve its accuracy and flexibility at interpreting human instructions over time, through normal everyday use. This novel approach aims to improve human interaction with intelligent multi-robot systems that aid the residents and visitors of a large, multi-use building. This fundamental research also supports computer-science education in the growing areas of natural-language processing, human-robot interaction, and machine learning, where there is significant national demand for knowledgeable personnel.The technical approach explored is a novel integration of learning techniques from three currently disparate areas: semantic parsing, spoken dialog management, and perceptual language grounding. Semantic parsing is the task of mapping natural language to a formal computer-interpretable language using compositional semantics based on syntactic linguistic structure. Dialog management concerns controlling multi-turn natural language interaction to aid comprehension and task completion. Perceptual grounding concerns associating words and phrases in language to objects, properties and relations in the world as perceived by the robot's sensors. Although there has been recent significant progress in each of these individual areas, no one has previously explored integrating them to support learning for human-robot communication through natural dialog. This exploratory research adapts and integrates techniques for semantic-parser learning using combinatory categorial grammar, dialog management using Partially Observable Markov Decision Processes, and multi-modal language grounding using both visual and haptic sensors, in order to develop a novel dialog system for communicating with robots that comprise the innovative Building Wide Intelligence system being developed at the University of Texas at Austin. The exploratory methods are evaluated using controlled experiments on a range of tasks using both on-line simulations and crowdsourced users, and natural user interaction with a mobile robot platform consisting of a wheeled Segway base and a Kinova robot arm.