Fluidity in simulated human-robot interaction with speech interfaces

考虑到我们现在面临的一些最大挑战，英国对能够与人类成功合作的交互式机器人的需求变得越来越重要，包括需要高价值制造业出口以在国际上进行经济竞争，可以处理危险废物和在危险环境中航行的机器人，以及用于社会护理和医疗援助的机器人解决方案，以满足我们的人口挑战。人机交互（HRI）的一个关键问题限制这种机器人更广泛使用的一个障碍是缺乏流动性。虽然最近在机器人视觉、运动、操纵和自动语音识别方面取得了重大进展，但最先进的HRI速度缓慢、费力且脆弱。与人与人之间互动的速度，流畅性和容错性形成了巨大的对比。FLUIDITY项目将开发技术，以监控、控制和提高具有语音理解能力的机器人的交互流畅性，使它们变得更加自然和有效。该项目还将通过开发一个工具包，用于在模拟的虚拟现实（VR）环境中构建和测试交互式机器人模型，为更广泛的机器人技术进行可扩展的HRI实验，HRI和自然语言处理（NLP）社区成为可能。该项目侧重于挑选和将操纵家用物品的机器人放置在用户可以发出命令的地方（例如，“把遥控器放在桌子上”），并适当地发出机器人当前动作的确认、纠正和修复（例如，“不，另一张桌子”），允许来自远程操作机器人模型的人类同盟者和自动系统的快速、自然的响应。至关重要的是，人类语音和机器人运动的适当重叠将被允许允许更像人类的过渡。该项目将把互动流动性和从误解中快速恢复与适当的修复机制放在互动机器人的核心，实现流畅交互的手段首先是适应口语理解（SLU）算法，这些算法不仅是逐词增量的，而且超越了更像人类的真实的-机器人在解释用户语音时的置信度的时间度量。在这些算法的基础上，将从成对的人类参与者中收集中介的Wizard-of-Oz数据，其中一个参与者邦联“向导”控制机器人模型和一个用户。从收集的视觉、音频和运动数据中，将构建SLU算法，该算法将逐词递增地返回最准确的用户意图，同时也是尽可能接近人类联盟反应时间的连续置信度度量。该项目还将通过实验不同的运动易读性模型，解决用户从机器人运动中感知机器人意图的问题。假设机器人运动的易读性可以在真实的时间中建模得越准确，交互的流动性就越大，因为用户的修复和确认可以在机器人的运动中更早地被适当地解释。SLU和易读性算法将被集成在端到端系统中，在该系统中交互流动性可以被控制，在VR环境中进行评估，并与真实世界的机器人模型进行比较。该项目将为VR环境的交互流动性和实际结果提供一个抽象的理论框架，在环境中收集的HRI数据集将公开用于基准测试，软件将是开源的，适用于其他机器人模型。

The need for interactive robots which can collaborate successfully with human beings is becoming important in the UK considering some of the biggest challenges we now face, including the need for high-value manufacturing exports to compete economically internationally, robots which can handle dangerous waste and navigate hazardous environments, and robotics solutions for social care and medical assistance to meet our demographic challenges.A key problem for human-robot interaction (HRI) with speech which limits the wider use of such robots is lack of fluidity. Although there have been significant recent advances in robot vision, motion, manipulation and automatic speech recognition, state-of-the-art HRI is slow, laboured and fragile. The contrast with the speed, fluency and error tolerance of human-human interaction is substantial. The FLUIDITY project will develop technology to monitor, control and increase the interaction fluidity of robots with speech understanding capabilities, such that they become more natural and efficient to interact with. The project will also address the difficulty of developing HRI models due to the time, logistics and cost of working with real-world robots by developing a toolkit for building and testing interactive robot models in a simulated Virtual Reality (VR) environment, making scalable HRI experiments for the wider robotics, HRI and natural language processing (NLP) communities possible.The project focusses on pick-and-place robots which manipulate household objects in view where users will utter commands (e.g. "put the remote control on the table") and issue confirmations and corrections and repairs of the robot's current actions appropriately (e.g. "no, the other table"), allowing rapid, natural responses from both a human confederate teleoperating the robot model and automatic systems. Crucially, appropriate overlap of human speech and robot motion will be permitted to allow more human-like transitions. The project will put interaction fluidity and the rapid recovery from misunderstanding with appropriate repair mechanisms at the heart of interactive robots, which will lead to improved user experience.The means for achieving fluid interaction will firstly be adaptation of Spoken Language Understanding (SLU) algorithms which are not only word-by-word incremental but go beyond that for more human-like real-time measures of confidence the robot has in its interpretation of the user's speech. For the basis of these algorithms, mediated Wizard-of-Oz data will be collected from pairs of human participants, with one participant confederate 'wizard' controlling the robot model and one user. From the visual, audio and motion data collected, SLU algorithms will be built which return the most accurate user intention incrementally word-by-word, but also a continuous measure of confidence corresponding as closely as possible to the reaction times of the human confederate.The project will also address user perception of the robot's intention from the robot's motion by experimenting with different models of motion legibility. The hypothesis is that the more accurately the legibility of the robot's motion can be modelled in real time, the greater the fluidity of interaction possible, as user repairs and confirmations can be interpreted appropriately earlier in the robot's motion.The SLU and legibility algorithms will be integrated in an end-to-end system where interaction fluidity can be controlled, with evaluation in both the VR environment and a comparison to a real-world robot model. The project will provide an abstract theoretical framework for interaction fluidity and practical outcomes of a VR environment, an HRI dataset collected in the environment which will be made publicly available for benchmarking, and software which will be open-source and adaptable for other robot models.