Combined biplanar X-ray and RGBD images 3D reconstruction of the musculoskeletal system for augmented reality clinical follow-up.

组合双平面 X 射线和 RGBD 图像对肌肉骨骼系统进行 3D 重建，用于增强现实临床随访。

• 批准号: RGPIN-2021-04293
• 负责人: Vazquez, Carlos
• 金额: $ 2.11万
• 依托单位: École de technologie supérieure
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=741940
• 关键词: Combined; biplanar; ray; RGBD; images

Imagine going to the hospital, taking a 1 min X-ray scan like the ones at the airport security and when you see the doctor, she/he has a tablet and sees your bones in the right positions in your body via a precise 3D augmented reality (AR) app. The doctor asks you to walk to identify the origin of that pain in your right knee and tells you immediately, while showing you your knee joint (the bones) superimposed on the image of your real knee, what the problem is. Or going to the ER with a fractured arm bone and seeing the precise depiction of the fracture inside your arm on the tablet, which the doctor is using to plan the best approach in putting the bone back together. This is the kind of system that we intend to make possible by building specialized 3D models of bones, developing algorithms to construct them from optical and X-ray images, and building an AR application to show the bones in the right place on a live video feed. Our research group at the LIO-ÉTS-CRCHUM has been working over the years building 3D models from low-dose biplanar X-ray images to facilitate the extraction of clinical parameters needed to diagnose conditions of the musculoskeletal system, e.g. scoliosis, or to analyze posture or walking disorders. We have developed expertise in the identification of bones, the rigid and non-rigid 2D/3D registration of 3D models and 2D images, pose estimation and AR applications development. We intend to expand our research efforts to build a multimodal acquisition system combining X-ray and image+depth (RGBD) imaging devices to construct a personalized 3D model of the patient that can be used later for real-time analysis of the musculoskeletal system. The proposed methodology is based on the use of deep learning approaches to identify the bones in the images and modify a combined statistical 3D model of the bones and the body envelope so it corresponds to the morphology of the patient. We will also develop a robust pose estimation method able to precisely place the bone images into an RGBD image of the patient. This process will be performed by learning, from a calibrated and synchronized pair of X-ray and RGBD images, the relationship between the shape of the envelope and the location of bones. An AR clinical application to show the model superimposed on the image of the patient and the relevant clinical parameters will be developed to show the potential applications of the system. Being a certified Living Lab, we include all the stakeholders to the research from the beginning. We count on the participation of clinical and industrial collaborators to ensure that the algorithms and methodologies that we develop are applicable in clinical routine and amenable to commercialization by medical equipment companies. We intend to transfer these promising technologies to industrial collaborators, to make them available to all doctors and allow faster and more accurate analysis of the musculoskeletal system for the benefit of the patients.

想象去医院,1分钟在机场x射线扫描的安全,当你看医生,她/他有一个平板电脑,看到你的骨骼在正确的位置在你的身体通过精确的3 d增强现实应用(AR)。医生问你走到确定的起源,疼痛立即你的右膝盖,告诉你,同时展示你的膝关节(骨头)叠加在你真正的形象膝盖,问题是什么。或者带着骨折的手臂去急诊室，在平板上看到手臂骨折的精确描述，医生用它来计划最好的方法把骨头接回去。我们打算通过建立专门的骨骼3D模型，开发从光学和x射线图像构建模型的算法，以及构建一个AR应用程序来在实时视频中显示骨骼的正确位置，从而使这种系统成为可能。我们在LIO-ÉTS-CRCHUM的研究小组多年来一直致力于从低剂量双平面x射线图像中构建3D模型，以方便提取诊断肌肉骨骼系统状况所需的临床参数，例如脊柱侧凸，或分析姿势或行走障碍。我们在骨骼识别，3D模型和2D图像的刚性和非刚性2D/3D注册，姿态估计和AR应用开发方面拥有专业知识。我们打算扩大我们的研究工作，建立一个结合x射线和图像+深度（RGBD）成像设备的多模态采集系统，构建一个个性化的患者3D模型，可用于肌肉骨骼系统的实时分析。所提出的方法是基于使用深度学习方法来识别图像中的骨骼，并修改骨骼和身体包膜的组合统计3D模型，使其与患者的形态学相对应。我们还将开发一种鲁棒的姿态估计方法，能够精确地将骨骼图像放入患者的RGBD图像中。这个过程将通过学习，从校准和同步的x射线和RGBD图像对，包膜的形状和骨头的位置之间的关系来执行。将开发AR临床应用程序，以显示叠加在患者图像上的模型和相关临床参数，以显示该系统的潜在应用。作为一个经过认证的生活实验室，我们从一开始就邀请所有利益相关者参与研究。我们依靠临床和工业合作者的参与，以确保我们开发的算法和方法适用于临床常规，并适合医疗设备公司的商业化。我们打算将这些有前途的技术转让给工业合作者，使所有医生都可以使用，并允许更快，更准确地分析肌肉骨骼系统，以造福患者。