Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training
Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training
复制标题
DOI：
10.3390/app12105159
复制
发表时间：
2022-05-01
影响因子：
2.7
通讯作者：
Niyaz, Quamar
中科院分区：
文献类型：
--
作者：
Estrada, John;Paheding, Sidike;Niyaz, Quamar
深度学习（DL）算法在目标检测任务中取得了显著的高性能。与此同时，增强现实（AR）技术正在改变我们工作和与人联系的方式。随着在线和混合学习的日益普及，我们提出了一个新的框架，通过结合上述技术来改善学生对电气工程实验室设备的学习体验。集成到AR应用程序中的DL供电自动对象检测组件旨在识别万用表、示波器、波发生器和电源等设备。使用TensorFlow的对象检测API实现了一个深度神经网络模型，即MobileNet-SSD v2，用于设备检测。当检测到某件装备时，屏幕上会显示相应的ar教程。所开发的设备检测模型的平均精度（mAP）为81.4%，平均召回率为85.3%。此外，为了演示所提出的框架的实际应用，我们开发了一个万用表教程，其中虚拟模型叠加在真实的万用表上。本教程包括图片和网络链接，以帮助用户更有效地学习。Unity3D游戏引擎被用作本教程的主要开发工具，用于集成DL和AR框架并创建沉浸式场景。提出的框架可以为工业和教育培训的AR和基于机器学习的框架提供有用的基础。
Deep learning (DL) algorithms have achieved significantly high performance in object detection tasks. At the same time, augmented reality (AR) techniques are transforming the ways that we work and connect with people. With the increasing popularity of online and hybrid learning, we propose a new framework for improving students' learning experiences with electrical engineering lab equipment by incorporating the abovementioned technologies. The DL powered automatic object detection component integrated into the AR application is designed to recognize equipment such as multimeter, oscilloscope, wave generator, and power supply. A deep neural network model, namely MobileNet-SSD v2, is implemented for equipment detection using TensorFlow's object detection API. When a piece of equipment is detected, the corresponding AR-based tutorial will be displayed on the screen. The mean average precision (mAP) of the developed equipment detection model is 81.4%, while the average recall of the model is 85.3%. Furthermore, to demonstrate practical application of the proposed framework, we develop a multimeter tutorial where virtual models are superimposed on real multimeters. The tutorial includes images and web links as well to help users learn more effectively. The Unity3D game engine is used as the primary development tool for this tutorial to integrate DL and AR frameworks and create immersive scenarios. The proposed framework can be a useful foundation for AR and machine-learning-based frameworks for industrial and educational training.