Blind Recognition of Touched Keys on Mobile Devices
Blind Recognition of Touched Keys on Mobile Devices
复制标题
DOI：
10.1145/2660267.2660288
复制
发表时间：
2014-11
期刊：
影响因子：
--
通讯作者：
Qinggang Yue;Zhen Ling;Xinwen Fu;Benyuan Liu;K. Ren;Wei Zhao
中科院分区：
文献类型：
--
作者：
Qinggang Yue;Zhen Ling;Xinwen Fu;Benyuan Liu;K. Ren;Wei Zhao
在本文中，我们介绍了一种新颖的基于计算机视觉的攻击，该攻击可以自动披露支持触摸的设备上的输入，而攻击者无法在受害者点击触摸屏的视频中看到任何文本或弹出框。我们仔细分析了指尖周围阴影的形成，应用光流、可变形零件模型（DPM）、k-means聚类等计算机视觉技术自动定位触摸点。然后应用平面单应性将估计的触摸点映射到软件键盘键的参考图像。密码识别是极具挑战性的，因为没有语言模型可以应用于正确的估计触摸键。我们的威胁模型是，网络摄像头、智能手机或谷歌Glass被用于在会议和类似的聚会场所等场景中进行隐形攻击。我们讨论了用一个手指敲击和用多个手指和两只手敲击的两种情况。进行了大量的实验来证明这种攻击的影响。每个字符（或每个数字）的识别率超过97%，而识别4字符密码的成功率超过90%。我们的工作是第一个自动盲目识别在移动设备触摸屏上输入的随机密码（或密码），成功率非常高。
In this paper, we introduce a novel computer vision based attack that automatically discloses inputs on a touch-enabled device while the attacker cannot see any text or popup in a video of the victim tapping on the touch screen. We carefully analyze the shadow formation around the fingertip, apply the optical flow, deformable part-based model (DPM), k-means clustering and other computer vision techniques to automatically locate the touched points. Planar homography is then applied to map the estimated touched points to a reference image of software keyboard keys. Recognition of passwords is extremely challenging given that no language model can be applied to correct estimated touched keys. Our threat model is that a webcam, smartphone or Google Glass is used for stealthy attack in scenarios such as conferences and similar gathering places. We address both cases of tapping with one finger and tapping with multiple fingers and two hands. Extensive experiments were performed to demonstrate the impact of this attack. The per-character (or per-digit) success rate is over 97% while the success rate of recognizing 4-character passcodes is more than 90%. Our work is the first to automatically and blindly recognize random passwords (or passcodes) typed on the touch screen of mobile devices with a very high success rate.