Makeup Lamps: Live Augmentation of Human Faces via Projection
Makeup Lamps: Live Augmentation of Human Faces via Projection
复制标题
DOI：
10.1111/cgf.13128
复制
发表时间：
2017-05
影响因子：
2.5
通讯作者：
Amit H. Bermano;Markus Billeter;D. Iwai;Anselm Grundhöfer
中科院分区：
文献类型：
--
作者：
Amit H. Bermano;Markus Billeter;D. Iwai;Anselm Grundhöfer
我们提出了第一个系统的人脸实时动态增强。使用基于投影仪的照明，我们在小说表演中改变了人类表演者的外观。实时增强的关键挑战是延迟-根据特定姿势生成图像，但在投影时显示在不同的面部配置上。因此，我们的系统旨在减少过程中每一步的延迟，从捕获到处理再到投影。使用红外照明，光学和计算对齐的高速摄像头检测面部方向以及表情。将估计的表情融合变形映射到低维空间上，并且通过自适应卡尔曼滤波来估计、平滑和预测面部运动和非刚性变形。最后，根据时间、全局位置和表情对预先计算的偏移纹理进行插值，生成所需的外观。我们通过优化的CPU和GPU原型评估了我们的系统，并成功地为不同的表演者和表演者展示了不同的面部游戏和运动速度。与现有方法相比，所提出的系统是第一种完全支持动态面部投影映射而不需要任何物理跟踪标记并结合面部表情的方法。
We propose the first system for live dynamic augmentation of human faces. Using projector‐based illumination, we alter the appearance of human performers during novel performances. The key challenge of live augmentation is latency — an image is generated according to a specific pose, but is displayed on a different facial configuration by the time it is projected. Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection. Using infrared illumination, an optically and computationally aligned high‐speed camera detects facial orientation as well as expression. The estimated expression blendshapes are mapped onto a lower dimensional space, and the facial motion and non‐rigid deformation are estimated, smoothed and predicted through adaptive Kalman filtering. Finally, the desired appearance is generated interpolating precomputed offset textures according to time, global position, and expression. We have evaluated our system through an optimized CPU and GPU prototype, and demonstrated successful low latency augmentation for different performers and performances with varying facial play and motion speed. In contrast to existing methods, the presented system is the first method which fully supports dynamic facial projection mapping without the requirement of any physical tracking markers and incorporates facial expressions.