Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display
Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display
复制标题
DOI：
10.1109/vr.2017.7892245
复制
发表时间：
2017-03
期刊：
影响因子：
--
通讯作者：
Katsuhiro Suzuki;Fumihiko Nakamura;Jiu Otsuka;Katsutoshi Masai;Yuta Itoh;Yuta Sugiura;M. Sugimoto
中科院分区：
文献类型：
--
作者：
Katsuhiro Suzuki;Fumihiko Nakamura;Jiu Otsuka;Katsutoshi Masai;Yuta Itoh;Yuta Sugiura;M. Sugimoto
我们提出了一种虚拟化身与头戴式显示器（HMD）用户之间的面部表情映射技术。头戴式显示器让人们享受身临其境的虚拟现实（VR）体验。虚拟化身可以是用户在虚拟环境中的代表。然而，虚拟化身的表情与HMD用户的表情同步是有限的。佩戴头戴式头盔的主要问题是用户脸部的很大一部分被遮挡，使得在基于头戴式头盔的虚拟环境中进行面部识别变得困难。为了克服这一问题，我们提出了一种使用反射式光电传感器的面部表情映射技术。头戴式头盔内的传感器测量传感器与用户面部之间的距离。使用五种基本面部表情（中性、快乐、愤怒、惊讶和悲伤）的距离值来训练神经网络来估计用户的面部表情。我们在识别面部表情方面达到了88%的总体准确率。我们的系统还可以通过使用回归技术，通过现有的化身实时再现面部表情的变化。因此，我们的系统能够估计和重建与用户情绪变化相对应的面部表情。
We propose a facial expression mapping technology between virtual avatars and Head-Mounted Display (HMD) users. HMD allow people to enjoy an immersive Virtual Reality (VR) experience. A virtual avatar can be a representative of the user in the virtual environment. However, the synchronization of the the virtual avatar's expressions with those of the HMD user is limited. The major problem of wearing an HMD is that a large portion of the user's face is occluded, making facial recognition difficult in an HMD-based virtual environment. To overcome this problem, we propose a facial expression mapping technology using retro-reflective photoelectric sensors. The sensors attached inside the HMD measures the distance between the sensors and the user's face. The distance values of five basic facial expressions (Neutral, Happy, Angry, Surprised, and Sad) are used for training the neural network to estimate the facial expression of a user. We achieved an overall accuracy of 88% in recognizing the facial expressions. Our system can also reproduce facial expression change in real-time through an existing avatar using regression. Consequently, our system enables estimation and reconstruction of facial expressions that correspond to the user's emotional changes.