Real-Time 3D Object Detection, Recognition and Presentation Using a Mobile Device for Assistive Navigation
Real-Time 3D Object Detection, Recognition and Presentation Using a Mobile Device for Assistive Navigation
复制标题
DOI：
10.1007/s42979-023-01881-3
复制
发表时间：
2023-07
期刊：
影响因子：
--
通讯作者：
Jin Chen;Zhigang Zhu
中科院分区：
文献类型：
--
作者：
Jin Chen;Zhigang Zhu
本文提出了一种集成的解决方案，三维物体检测，识别和演示，以增加可访问性的各种用户群体在室内区域通过移动的应用程序。该系统有三个主要组成部分：3D物体检测模块，物体跟踪和更新模块，以及语音和AR增强界面。3D对象检测模块由预训练的2D对象检测器和3D边界框估计方法组成，用于检测每个相机帧中对象的3D姿态和大小。该模块可以很容易地适应各种2D对象检测器（例如，YOLO、SSD、mask RCNN），基于所请求的任务和运行时间的要求以及3D检测结果的细节。它可以在云服务器或移动的应用程序上运行。对象跟踪和更新模块通过将2D跟踪结果转换为3D结果来最大限度地减少长期环境扫描的计算能力。语音和AR增强界面集成了ARKit和SiriKit，提供语音交互和AR可视化，以改善不同用户群体的信息传递。该系统可以与现有的应用程序集成，特别是辅助导航，以提高盲人或视力低下者的旅行安全性，并改善自闭症谱系障碍患者的社交互动。此外，它还可以用于其他应用的环境的3D重建。我们的初步测试结果的对象检测评估和实时系统的性能，以验证所提出的系统。
This paper presents an integrated solution for 3D object detection, recognition, and presentation to increase accessibility for various user groups in indoor areas through a mobile application. The system has three major components: a 3D object detection module, an object tracking and update module, and a voice and AR-enhanced interface. The 3D object detection module consists of pre-trained 2D object detectors and 3D bounding box estimation methods to detect the 3D poses and sizes of the objects in each camera frame. This module can easily adapt to various 2D object detectors (e.g., YOLO, SSD, mask RCNN) based on the requested task and requirements of the run time and details for the 3D detection result. It can run on a cloud server or mobile application. The object tracking and update module minimizes the computational power for long-term environment scanning by converting 2D tracking results into 3D results. The voice and AR-enhanced interface integrates ARKit and SiriKit to provide voice interaction and AR visualization to improve information delivery for different user groups. The system can be integrated with existing applications, especially assistive navigation, to increase travel safety for people who are blind or have low vision and improve social interaction for individuals with autism spectrum disorder. In addition, it can potentially be used for 3D reconstruction of the environment for other applications. Our preliminary test results for the object detection evaluation and real-time system performance are provided to validate the proposed system.