Fusing and refining convolutional neural network models for assembly action recognition in smart manufacturing
Fusing and refining convolutional neural network models for assembly action recognition in smart manufacturing
复制标题
DOI：
10.1177/0954406220931547
复制
发表时间：
2020-06-11
影响因子：
2
通讯作者：
Leu, Ming C.
中科院分区：
文献类型：
--
作者：
Al-Amin, Md.;Qin, Ruwen;Leu, Ming C.
装配在制造中至关重要。能够实时支持工人最大限度地发挥他们对装配的积极贡献是制造商的巨大兴趣。人类动作识别是一种自动分析和理解工人动作的方法，以支持对工人的实时帮助并促进工人与机器的协作。组装动作与动作识别文献中已深入研究的活动不同。装配工人采取的动作是复杂的、多变的，并且可能涉及非常精细的动作。因此，识别装配动作仍然是一项具有挑战性的任务。本文提出仅使用两个可穿戴设备，分别捕获工人每只手的惯性测量单元数据。然后，使用两个惯性测量单元数据源独立训练两个具有相同架构的卷积神经网络模型，以分别识别装配工人的右手和左手动作。由于两只手经常协作进行组装操作，因此两个卷积神经网络模型的分类结果被融合以产生最终的动作识别结果。实施迁移学习以使动作识别模型适应其数据尚未包含在用于训练模型的数据集中的受试者。组装 Bukito 三维打印机的一项操作由七个动作组成，用于演示所提出方法的实施和评估。研究结果表明，所提出的方法有效提高了行动层面和主体层面的预测准确性。本文的工作为构建先进的动作识别系统（例如基于多模态传感器的动作识别）奠定了基础。
Assembly carries paramount importance in manufacturing. Being able to support workers in real time to maximize their positive contributions to assembly is a tremendous interest of manufacturers. Human action recognition has been a way to automatically analyze and understand worker actions to support real-time assistance for workers and facilitate worker-machine collaboration. Assembly actions are distinct from activities that have been well studied in the action recognition literature. Actions taken by assembly workers are intricate, variable, and may involve very fine motions. Therefore, recognizing assembly actions remains a challenging task. This paper proposes to simply use only two wearable devices that respectively capture the inertial measurement unit data of each hand of workers. Then, two convolutional neural network models with an identical architecture are independently trained using the two sources of inertial measurement unit data to respectively recognize the right-hand and the left-hand actions of an assembly worker. Classification results of the two convolutional neural network models are fused to yield a final action recognition result because the two hands often collaborate in assembling operations. Transfer learning is implemented to adapt the action recognition models to subjects whose data have not been included in dataset for training the models. One operation in assembling a Bukito three-dimensional printer, which is composed of seven actions, is used to demonstrate the implementation and assessment of the proposed method. Results from the study have demonstrated that the proposed approach effectively improves the prediction accuracy at both the action level and the subject level. Work of the paper builds a foundation for building advanced action recognition systems such as multimodal sensor-based action recognition.