An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration
An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration
复制标题
DOI：
10.1109/ro-man57019.2023.10309444
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Afagh Mehri Shervedani;S. Li;Natawut Monaikul;Bahareh Abbasi;Barbara Maria Di Eugenio;M. Žefran
中科院分区：
文献类型：
--
作者：
Afagh Mehri Shervedani;S. Li;Natawut Monaikul;Bahareh Abbasi;Barbara Maria Di Eugenio;M. Žefran
本文提出了一种基于神经网络的用户模拟器，可以提供多模式交互环境，用于在涉及多种通信模式的协作任务中训练强化学习（RL）代理。该模拟器在现有的老年人在家语料库上进行训练，并适应多种形式，如语言，指向手势和触觉-明示动作。本文还提出了一种新的多模态数据增强方法，该方法解决了由于收集人类演示的昂贵和耗时性而使用有限数据集的挑战。总体而言，该研究强调了使用RL和多模式用户模拟器开发和改进家用辅助机器人的潜力。
This paper proposes a neural network-based user simulator that can provide a multimodal interactive environment for training Reinforcement Learning (RL) agents in collaborative tasks involving multiple modes of communication. The simulator is trained on the existing ELDERLY-AT-HOME corpus and accommodates multiple modalities such as language, pointing gestures, and haptic-ostensive actions. The paper also presents a novel multimodal data augmentation approach, which addresses the challenge of using a limited dataset due to the expensive and time-consuming nature of collecting human demonstrations. Overall, the study highlights the potential for using RL and multimodal user simulators in developing and improving domestic assistive robots.