Facial Performance Sensing Head-Mounted Display
Facial Performance Sensing Head-Mounted Display
复制标题
DOI：
10.1145/2766939
复制
发表时间：
2015-08-01
影响因子：
6.2
通讯作者：
Ma, Chongyang
中科院分区：
文献类型：
--
作者：
Li, Hao;Trutoiu, Laura;Ma, Chongyang
目前还没有解决方案能够实现佩戴头戴式显示器（HMD）的虚拟现实（VR）用户之间的直接面对面交互。主要的挑战是，头戴式耳机遮挡了用户面部的很大一部分，妨碍了传统技术的有效面部捕捉。为了推动虚拟现实成为下一代通信平台，我们开发了一种新型头戴式显示器，可以实时实现3D面部表演驱动的动画。我们的可穿戴系统使用超薄柔性电子材料，安装在耳机的泡沫衬垫上，以测量与上面部表情相对应的表面应变信号。这些应变信号与头戴式RGB-D摄像头相结合，以增强嘴部区域的跟踪，并解决HMD放置不准确的问题。为了将输入信号映射到3D人脸模型，我们为每个人执行单实例离线训练会话。为了可重复使用和准确的在线操作，我们提出了一个简短的校准步骤，在每次使用之前重新调整映射的高斯混合分布。由此产生的动画在视觉上与尖端的深度传感器驱动的面部表现捕捉系统不相上下，因此适合虚拟世界中的社交互动。
There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.