Effect of Kinematics and Fluency in Adversarial Synthetic Data Generation for ASL Recognition With RF Sensors
Effect of Kinematics and Fluency in Adversarial Synthetic Data Generation for ASL Recognition With RF Sensors
复制标题
DOI：
10.1109/taes.2021.3139848
复制
发表时间：
2021-12
影响因子：
4.4
通讯作者：
Mohammad Mahbubur Rahman;E. Malaia;A. Gurbuz;Darrin J. Griffin;Chris S. Crawford;S. Gurbuz
中科院分区：
文献类型：
--
作者：
Mohammad Mahbubur Rahman;E. Malaia;A. Gurbuz;Darrin J. Griffin;Chris S. Crawford;S. Gurbuz
射频传感器最近被提出作为一种新的模式，手语处理技术。它们是非接触的，在黑暗中有效的，并通过利用微多普勒效应获得直接测量的签署运动学。首先，这项工作提供了一个深入的比较研究的运动学特性签署的RF传感器测量流利的ASL用户和听力模仿签名。其次，由于利用深度学习的ASL识别技术需要大量的训练数据，这项工作研究了手势运动学和主体流畅性对数据合成的对抗学习技术的影响。提出了以下两种不同的合成训练数据生成方法：1）对抗域自适应，以最大限度地减少模仿签名和流畅签名数据之间的差异; 2）运动学约束的生成对抗网络，用于准确合成RF签名。结果表明，模仿签名和流畅的签名之间的运动学差异是如此显着，直接从流畅的RF签名合成的数据上的训练提供了更大的性能（93%的前5名的准确性）比模仿签名（88%的前5名的准确性）的适应时产生的分类100 ASL标志。
RF sensors have been recently proposed as a new modality for sign language processing technology. They are noncontact, effective in the dark, and acquire a direct measurement of signing kinematic via exploitation of the micro-Doppler effect. First, this work provides an in depth comparative examination of the kinematic properties of signing as measured by RF sensors for both fluent ASL users and hearing imitation signers. Second, as ASL recognition techniques utilizing deep learning requires a large amount of training data, this work examines the effect of signing kinematics and subject fluency on adversarial learning techniques for data synthesis. The following two different approaches for the synthetic training data generation are proposed: 1) adversarial domain adaptation to minimize the differences between imitation signing and fluent signing data and 2) kinematically-constrained generative adversarial networks for accurate synthesis of RF signing signatures. The results show that the kinematic discrepancies between imitation signing and fluent signing are so significant that training on data directly synthesized from fluent RF signers offers greater performance (93% top-5 accuracy) than that produced by adaptation of imitation signing (88% top-5 accuracy) when classifying 100 ASL signs.