2D-3D Reconstruction for internal organs using Deep Learning Techniques

本研究的目的是发展一种数据驱动的方法，可以从2D图像重建3D数据。典型的应用是从X射线图像重建3D CT数据。如果我们能够实现这个目标，我们可以在手术中更准确地定位肿瘤。为了实现这个目标，我们提出了两种不同的方法来进行2D-3D重建。首先，我们从比正常情况下更少的视图重建三维CT数据，然后减少视图的数量。在Z轴上实现了从少到多的重建，最近发表了结果。我们试图将视图的数量减少到近似2D。然而，结果并不那么好。因此，我们提出了第二种方法，我们实现了现有的神经网络X2 CT-GAN。该模型可以从两幅正交的X射线图像重建三维CT数据。然而，精度不够好（SSIM约为0.62）。虽然我们使用数据增强来提高其性能，但SSIM约为0.74，这对于临床应用来说仍然太低。这一次，我们利用了训练CT数据的3D拓扑，并将3D拓扑特征添加到X2 CT-GAN网络中。在我们的模型中，系统可以从两个不一定正交的X射线图像重建3D CT数据。我们将我们的方法与原始的X2 CT-GAN进行了比较，发现我们的方法具有更好的性能。目前，我们正在实施更多的比较实验，以证明所提出的模型的有效性。

The purpose of this research is to develop a data-driven approach which can reconstruct 3D data from 2D images. A typical application is to reconstruct 3D CT data from x-ray images. If we can realize this goal, we can locate the tumors much more accurately than the usual case during a surgery.To fulfill the goal, we have proposed two different approaches for the 2D-3D reconstruction. First, we reconstructed the 3D CT data from fewer views than normal cases and then reduced the number of views. From-less-to-more reconstruction was implemented in Z axis, and the results were published recently. We tried to reduce the number of views to approximate 2D. However, the results were not so good.Therefore, we proposed the second approach where we implemented an existing neural network X2CT-GAN. This model can reconstruct 3D CT data from two orthogonal X-ray images. However, the accuracy is not good enough (SSIM is about 0.62). Although we used data augmentation to improve its performance, the SSIM is about 0.74, which is still too low for clinic applications. This time, we exploited the 3D topology of the training CT data, and added the 3D topology features to the X2CT-GAN network. In our model, the system can reconstruct 3D CT data from two x-ray images which are not necessarily the orthogonal ones.We compared our approach with the original X2CT-GAN, and found our method achieved better performance. Currently, we are implementing more comparison experiments to demonstrate the effectiveness of the proposed model.