Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors
Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors
复制标题
DOI：
10.1145/3447993.3483272
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Cong Shi;Xiangyu Xu;Tianfang Zhang;Pa Walker;Yi Wu;Jian Liu;Nitesh Saxena;Yingying Chen;Jiadi Yu
中科院分区：
文献类型：
--
作者：
Cong Shi;Xiangyu Xu;Tianfang Zhang;Pa Walker;Yi Wu;Jian Liu;Nitesh Saxena;Yingying Chen;Jiadi Yu
增强现实/虚拟现实（AR/VR）已经从3D沉浸式游戏扩展到更广泛的应用，如购物，旅游，教育。最近，从手持控制器主导的交互到通过语音界面的耳机主导的交互已经发生了很大的转变。在这项工作中，我们展示了在用户佩戴面部安装的AR/VR设备时使用语音界面的严重隐私风险。具体来说，我们设计了一种窃听攻击，Face-Mic，它利用AR/VR耳机中的零权限运动传感器捕获的与语音相关的微妙面部动态，从真人语音中推断出高度敏感的信息，包括说话者的性别、身份和语音内容。Face-Mic基于一个关键的洞察力，即AR/VR耳机紧密地安装在用户的脸上，允许耳机上的潜在恶意应用程序在佩戴者说话时捕获潜在的面部动态，包括面部肌肉的运动和骨传振动，这些振动编码私人生物特征和语音特征。为了减轻身体动作的影响，我们开发了一种信号源分离技术，以识别和分离语音相关的面部动态从其他类型的身体动作。我们进一步提取代表性的功能方面的两种类型的面部动态。我们成功地证明了通过AR/VR耳机的隐私泄露，通过开发基于深度学习的框架来获取用户的性别/身份并提取语音信息。使用四种主流VR头显进行的大量实验验证了Face-Mic的通用性、有效性和高准确性。
Augmented reality/virtual reality (AR/VR) has extended beyond 3D immersive gaming to a broader array of applications, such as shopping, tourism, education. And recently there has been a large shift from handheld-controller dominated interactions to headset-dominated interactions via voice interfaces. In this work, we show a serious privacy risk of using voice interfaces while the user is wearing the face-mounted AR/VR devices. Specifically, we design an eavesdropping attack, Face-Mic, which leverages speech-associated subtle facial dynamics captured by zero-permission motion sensors in AR/VR headsets to infer highly sensitive information from live human speech, including speaker gender, identity, and speech content. Face-Mic is grounded on a key insight that AR/VR headsets are closely mounted on the user's face, allowing a potentially malicious app on the headset to capture underlying facial dynamics as the wearer speaks, including movements of facial muscles and bone-borne vibrations, which encode private biometrics and speech characteristics. To mitigate the impacts of body movements, we develop a signal source separation technique to identify and separate the speech-associated facial dynamics from other types of body movements. We further extract representative features with respect to the two types of facial dynamics. We successfully demonstrate the privacy leakage through AR/VR headsets by deriving the user's gender/identity and extracting speech information via the development of a deep learning-based framework. Extensive experiments using four mainstream VR headsets validate the generalizability, effectiveness, and high accuracy of Face-Mic.