Combined biplanar X-ray and RGBD images 3D reconstruction of the musculoskeletal system for augmented reality clinical follow-up.

点击翻译按钮获取中文摘要

Imagine going to the hospital, taking a 1 min X-ray scan like the ones at the airport security and when you see the doctor, she/he has a tablet and sees your bones in the right positions in your body via a precise 3D augmented reality (AR) app. The doctor asks you to walk to identify the origin of that pain in your right knee and tells you immediately, while showing you your knee joint (the bones) superimposed on the image of your real knee, what the problem is. Or going to the ER with a fractured arm bone and seeing the precise depiction of the fracture inside your arm on the tablet, which the doctor is using to plan the best approach in putting the bone back together. This is the kind of system that we intend to make possible by building specialized 3D models of bones, developing algorithms to construct them from optical and X-ray images, and building an AR application to show the bones in the right place on a live video feed. Our research group at the LIO-ÉTS-CRCHUM has been working over the years building 3D models from low-dose biplanar X-ray images to facilitate the extraction of clinical parameters needed to diagnose conditions of the musculoskeletal system, e.g. scoliosis, or to analyze posture or walking disorders. We have developed expertise in the identification of bones, the rigid and non-rigid 2D/3D registration of 3D models and 2D images, pose estimation and AR applications development. We intend to expand our research efforts to build a multimodal acquisition system combining X-ray and image+depth (RGBD) imaging devices to construct a personalized 3D model of the patient that can be used later for real-time analysis of the musculoskeletal system. The proposed methodology is based on the use of deep learning approaches to identify the bones in the images and modify a combined statistical 3D model of the bones and the body envelope so it corresponds to the morphology of the patient. We will also develop a robust pose estimation method able to precisely place the bone images into an RGBD image of the patient. This process will be performed by learning, from a calibrated and synchronized pair of X-ray and RGBD images, the relationship between the shape of the envelope and the location of bones. An AR clinical application to show the model superimposed on the image of the patient and the relevant clinical parameters will be developed to show the potential applications of the system. Being a certified Living Lab, we include all the stakeholders to the research from the beginning. We count on the participation of clinical and industrial collaborators to ensure that the algorithms and methodologies that we develop are applicable in clinical routine and amenable to commercialization by medical equipment companies. We intend to transfer these promising technologies to industrial collaborators, to make them available to all doctors and allow faster and more accurate analysis of the musculoskeletal system for the benefit of the patients.