NRI: Collaborative Research: RobotSLANG: Simultaneous Localization, Mapping, and Language Acquisition

人类和机器人都迫切需要在新环境中导航以执行日常任务。 父母和孩子可能正在参观大学校园;机器人可能在建筑物倒塌后寻找幸存者。 在这两个机构的教师的合作中，PI设想人类和机器人合作伙伴分享共同的感知语言经验，并在日常任务中合作，如清洁工作和家庭护理，以及紧急响应或搜索和救援等关键任务。 但是，尽管地图和导航现在对于移动的机器人来说是司空见惯的，但当考虑人类与机器人协作完成甚至是简单的任务时，人们都面临着一个关键的障碍：机器人和人类没有共同的语言。 人类语言中有丰富的语言元素来描述我们的空间环境，其中的物体和地点，以及通过它的导航路径（例如，“沿着走廊走，进入右边的第三个门。"). 另一方面，机器人居住在一个由被占用和未被占用的离散网格单元组成的度量世界中，其中大多数对象都没有意义（语义）。 PI在该项目中的目标是通过将同时定位和映射（SLAM）的问题与语言习得问题结合起来来克服这一限制，以便使机器人能够学习用英语与人交流导航任务。 PI将通过模仿同名真人秀节目的Amazing Race挑战问题来激发科学界对这一新研究领域的兴趣，该挑战问题将机器人和人类机器人团队置于未知环境中，并要求他们尽快完成特定任务。 其他外展活动将包括参观K-12学校并进行示范。 这项工作将集中在同步本地化，映射和语言习得，一个领域的调查，仍然没有触及。 关键的原则是，语义被制定为成本函数，这反过来又指定了许多变量，包括捕捉感官输入，语言，环境地图，和机器人电机控制的联合分布。 成本函数和联合分布支持多种形式的标准推理，例如命令跟随。 更重要的是，它们支持多变量集合上的多方向推理，如同时映射和语言解释。 在这个创新的基于多变量优化的框架内，PI计划了一个全面的实验方案，包括具有挑战性环境的合成和真实世界数据集，将自然语言的语义建立在现实视觉世界和机器人电机控制的空间地图中，当沿着沿着特定路径导航或到达特定目的地时，（可能是新颖的）环境，不仅在几何意义上，而且还与语言基础映射到这些特定的路径和目的地。 语言方法是组合的，并使用名词（对象/地点）和介词（它们之间的关系）的空间接地表示。 这些空间接地表示将在映射的上下文中建模。 此外，PI将考虑现实环境并根据联合模型调整其视觉模型。 PI知道没有其他工作联合模型映射，视觉和语言习得。

Humans and robots alike have a critical need to navigate through new environments to carry out everyday tasks. A parent and child may be touring a college campus; a robot may be searching for survivors after a building has collapsed. In this collaboration by faculty at two institutions, the PIs envision human and robotic partners sharing common perceptual-linguistic experiences and cooperating in mundane tasks like janitorial work and home care as well as in critical tasks like emergency response or search-and-rescue. But while mapping and navigation are now commonplace for mobile robots, when considering human-robot collaboration for even simple tasks one is confronted by a critical barrier: robots and people do not share a common language. Human language is rich in linguistic elements for describing our spatial environment, the objects and places within it, and navigable paths through it (e.g., "go down the hallway and enter the third door on the right."). Robots, on the other hand, inhabit a metric world of occupied and unoccupied discretized grid cells, wherein most objects are devoid of meaning (semantics). The PIs' goal in this project is to overcome this limitation by conjoining the well understood problem of simultaneous localization and mapping (SLAM) with that of language acquisition, in order to enable robots to learn to communicate with people in English about navigation tasks. The PIs will spur interest in this novel research area within the scientific community by means of an Amazing Race challenge problem modeled after the reality television show of the same name, which will place robots and human-robot teams in unknown environments and charge them with completing a specific task as quickly as possible. Other outreach activities will include visits to K-12 schools with demonstrations. This work will focus on simultaneous localization, mapping, and language acquisition, a field of inquiry that remains untouched. The crucial principles are that semantics are formulated as a cost function, which in turn specifies a joint distribution over many variables including those capturing sensory input, language, the environment map, and robot motor control. The cost function and joint distribution support standard inference of many forms, such as command following. More importantly, they support multidirectional inference over multiple variable sets jointly, such as simultaneous mapping and language interpretation. Within this innovative multivariate optimization-based framework, the PIs plan a thorough experimental regimen including both synthetic and real-world datasets of challenging environments, grounding the semantics of natural language in spatial maps of the realistic visual world and robot motor control, while navigating along particular paths or to arrive at particular destinations in (possibly novel) environments that are mapped not only in a geometric sense but also with linguistic underpinning to these particular paths and destinations. The language approach is compositional and uses spatially-grounded representations of nouns (objects/places) and prepositions (relations between them). These spatially-grounded representations will be modeled in the context of mapping. Furthermore, the PIs will consider realistic environments and adapt visual models thereof according to the joint model. The PIs are aware of no other work that jointly models mapping, vision, and language acquisition.