3D Localization of Hand Acupoints Using Hand Geometry and Landmark Points Based on RGB-D CNN Fusion
3D Localization of Hand Acupoints Using Hand Geometry and Landmark Points Based on RGB-D CNN Fusion
复制标题
DOI：
10.1007/s10439-022-02986-1
复制
发表时间：
2022-06
影响因子：
3.8
通讯作者：
Danish Masood;Jiang Qi
中科院分区：
文献类型：
--
作者：
Danish Masood;Jiang Qi
穴位刺激已被证明对康复和预防治疗具有重要意义。艾灸作为一种穴位疗法，一直以来主要由实践者依靠人工对穴位进行定位和定位，导致由于人为误差导致准确性的差异。使用深度学习技术自动检测穴位的发展已被证明在一定程度上解决了这个问题。但目前的方法缺乏基于深度的定位，因此仅限于二维（2D）定位。在本研究中，提出了一种基于RGB和深度卷积神经网络（CNN）融合的三维穴位定位方法。本研究旨在解决实时三维穴位定位的挑战，为机器人控制艾灸提供指导。第一步，对3D传感器（Kinect v1）进行校准，计算变换矩阵，将深度数据投影到RGB域。其次，采用RGB-CNN和depth-CNN的融合，得到三维定位；最后，将三维坐标输入机械手进行人工控制的艾灸治疗。此外，构建了一个由手部RGB图像和深度图像组成的三维穴位数据集，用于训练、验证和测试网络。该网络能够定位5组穴位，平均定位误差小于0.09。进一步的实验验证了该方法的有效性，为自动艾灸机器人的开发奠定了基础。
Acupoint stimulation has proven to be of significant importance for rehabilitation and preventive therapy. Moxibustion, a kind of acupoint therapy, has mainly been performed by practitioners relying on manual localization and positioning of acupoints, leading to variance in the accuracy owing to human error. Developments in the automatic detection of acupoints using deep learning techniques have proven to somewhat tackle the problem. But the current methods lack depth-based localization and are thus confined to two-dimensional (2D) localization. In this research, a new approach towards 3D acupoint localization is introduced, based on a fusion of RGB and depth convolutional neural networks (CNN) to guide the manipulator. This research aims to tackle the challenge of real-time 3D acupoint localization in order to provide guidance for robot-controlled moxibustion. In the first step, the 3D sensor (Kinect v1) is calibrated and transformation matrix is computed to project the depth data into the RGB domain. Secondly, a fusion of RGB-CNN and depth-CNN is employed, in order to obtain 3D localization. Lastly, 3D coordinates are fed to the manipulator to perform artificially controlled moxibustion therapy. Furthermore, a 3D acupoint dataset consisting of RGB and depth images of hands, is constructed to train, validate and test the network. The network was able to localize 5 sets of acupoints with an average localization error of less than 0.09. Further experiments prove the efficacy of the approach and lay grounds for development of automatic moxibustion robots.