Combined biplanar X-ray and RGBD images 3D reconstruction of the musculoskeletal system for augmented reality clinical follow-up.

组合双平面 X 射线和 RGBD 图像对肌肉骨骼系统进行 3D 重建，用于增强现实临床随访。

• 批准号: RGPIN-2021-04293
• 负责人: Vazquez, Carlos
• 金额: $ 2.11万
• 依托单位: École de technologie supérieure
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=754254
• 关键词: Combined; biplanar; ray; RGBD; images

Imagine going to the hospital, taking a 1 min X-ray scan like the ones at the airport security and when you see the doctor, she/he has a tablet and sees your bones in the right positions in your body via a precise 3D augmented reality (AR) app. The doctor asks you to walk to identify the origin of that pain in your right knee and tells you immediately, while showing you your knee joint (the bones) superimposed on the image of your real knee, what the problem is. Or going to the ER with a fractured arm bone and seeing the precise depiction of the fracture inside your arm on the tablet, which the doctor is using to plan the best approach in putting the bone back together. This is the kind of system that we intend to make possible by building specialized 3D models of bones, developing algorithms to construct them from optical and X-ray images, and building an AR application to show the bones in the right place on a live video feed. Our research group at the LIO-ÉTS-CRCHUM has been working over the years building 3D models from low-dose biplanar X-ray images to facilitate the extraction of clinical parameters needed to diagnose conditions of the musculoskeletal system, e.g. scoliosis, or to analyze posture or walking disorders. We have developed expertise in the identification of bones, the rigid and non-rigid 2D/3D registration of 3D models and 2D images, pose estimation and AR applications development. We intend to expand our research efforts to build a multimodal acquisition system combining X-ray and image+depth (RGBD) imaging devices to construct a personalized 3D model of the patient that can be used later for real-time analysis of the musculoskeletal system. The proposed methodology is based on the use of deep learning approaches to identify the bones in the images and modify a combined statistical 3D model of the bones and the body envelope so it corresponds to the morphology of the patient. We will also develop a robust pose estimation method able to precisely place the bone images into an RGBD image of the patient. This process will be performed by learning, from a calibrated and synchronized pair of X-ray and RGBD images, the relationship between the shape of the envelope and the location of bones. An AR clinical application to show the model superimposed on the image of the patient and the relevant clinical parameters will be developed to show the potential applications of the system. Being a certified Living Lab, we include all the stakeholders to the research from the beginning. We count on the participation of clinical and industrial collaborators to ensure that the algorithms and methodologies that we develop are applicable in clinical routine and amenable to commercialization by medical equipment companies. We intend to transfer these promising technologies to industrial collaborators, to make them available to all doctors and allow faster and more accurate analysis of the musculoskeletal system for the benefit of the patients.

想象一下，你去医院，像在机场安检时那样做一分钟的X光扫描，当你去看医生时，他/她会拿着平板电脑，通过精确的3D增强现实（AR）应用程序看到你身体正确位置的骨骼。医生会让你走路，以确定右膝疼痛的来源，并立即告诉你，同时向你展示你的膝关节（骨头）叠加在你的真实的膝盖的图像上，问题是什么。或者带着骨折的臂骨去急诊室，在平板电脑上看到手臂内部骨折的精确描述，医生正在用它来计划将骨头重新组合在一起的最佳方法。这是我们打算通过构建骨骼的专门3D模型，开发从光学和X射线图像构建它们的算法，并构建一个AR应用程序来在实时视频馈送的正确位置显示骨骼来实现的系统。我们在LIO-ÉTS-CRCHUM的研究小组多年来一直致力于从低剂量双平面X射线图像构建3D模型，以促进诊断肌肉骨骼系统状况所需的临床参数的提取，例如脊柱侧凸，或分析姿势或行走障碍。我们在骨骼识别、3D模型和2D图像的刚性和非刚性2D/3D配准、姿态估计和AR应用程序开发方面积累了专业知识。我们打算扩大我们的研究工作，建立一个多模式采集系统结合X射线和图像+深度（RGBD）成像设备，以构建个性化的三维模型的病人，可用于以后的肌肉骨骼系统的实时分析。所提出的方法基于使用深度学习方法来识别图像中的骨骼，并修改骨骼和身体包络的组合统计3D模型，使其与患者的形态相对应。我们还将开发一种鲁棒的姿态估计方法，能够精确地将骨骼图像放置到患者的RGBD图像中。该过程将通过从一对经校准和同步的X射线和RGBD图像中学习包络形状与骨骼位置之间的关系来执行。将开发一个AR临床应用程序，显示叠加在患者图像上的模型和相关临床参数，以显示该系统的潜在应用。作为一个认证的生活实验室，我们从一开始就包括所有利益相关者的研究。我们依靠临床和工业合作者的参与，以确保我们开发的算法和方法适用于临床常规，并适合医疗设备公司的商业化。我们打算将这些有前途的技术转让给工业合作者，使所有医生都能使用这些技术，并允许更快、更准确地分析肌肉骨骼系统，以造福患者。