Action Recognition From a Single Coded Image
Action Recognition From a Single Coded Image
复制标题
DOI：
10.1109/tpami.2022.3196350
复制
发表时间：
2022-08
影响因子：
23.6
通讯作者：
Sudhakar Kumawat;Tadashi Okawara;Michitaka Yoshida;H. Nagahara;Y. Yagi
中科院分区：
文献类型：
--
作者：
Sudhakar Kumawat;Tadashi Okawara;Michitaka Yoshida;H. Nagahara;Y. Yagi
深度卷积神经网络(CNN)在基于视频的人类行为识别任务上取得了前所未有的成功，前提是有良好的分辨率视频和资源来开发和部署复杂的模型。不幸的是，相机系统和识别模型上的某些预算和环境限制可能无法适应这些假设，并要求降低其复杂性。为了缓解这些问题，我们引入了一种深度传感解决方案，可以直接从编码曝光图像中识别人体行为。我们的深度传感解决方案包括一个基于CNN的二进制编码器网络，它模拟使用编码曝光相机捕获动态场景的编码曝光图像，然后是2D CNN，用于识别捕获的编码曝光图像中的人类行为。此外，我们提出了一种新的知识提取框架来联合训练编码者和动作识别模型，并表明所提出的训练方法在${2}$2-v2，Kinetics-400和UCF-101数据集上的动作识别准确率分别比以前的方法提高了6.2%，2.9%和7.9%。最后，我们利用LCoS构建了一个编码曝光相机的原型，以验证我们的深度传感方案的可行性。我们对原型相机的评估结果与仿真结果一致。
The unprecedented success of deep convolutional neural networks (CNN) on the task of video-based human action recognition assumes the availability of good resolution videos and resources to develop and deploy complex models. Unfortunately, certain budgetary and environmental constraints on the camera system and the recognition model may not be able to accommodate these assumptions and require reducing their complexity. To alleviate these issues, we introduce a deep sensing solution to directly recognize human actions from coded exposure images. Our deep sensing solution consists of a binary CNN-based encoder network that emulates the capturing of a coded exposure image of a dynamic scene using a coded exposure camera, followed by a 2D CNN for recognizing human action in the captured coded exposure image. Furthermore, we propose a novel knowledge distillation framework to jointly train the encoder and the action recognition model and show that the proposed training approach improves the action recognition accuracy by an absolute margin of 6.2%, 2.9%, and 7.9% on Something$^{2}$2-v2, Kinetics-400, and UCF-101 datasets, respectively, in comparison to our previous approach. Finally, we built a prototype coded exposure camera using LCoS to validate the feasibility of our deep sensing solution. Our evaluation of the prototype camera show results that are consistent with the simulation results.