DeepOrganNet: On-the-Fly Reconstruction and Visualization of 3D / 4D Lung Models from Single-View Projections by Deep Deformation Network
DeepOrganNet: On-the-Fly Reconstruction and Visualization of 3D / 4D Lung Models from Single-View Projections by Deep Deformation Network
复制标题
DOI：
10.1109/tvcg.2019.2934369
复制
发表时间：
2020-01-01
影响因子：
5.2
通讯作者：
Hua, Jing
中科院分区：
文献类型：
--
作者：
Wang, Yifan;Zhong, Zichun;Hua, Jing
本文介绍了一种基于深度神经网络的方法，即DeepOrganNet，用于从复杂背景的单视医学图像中实时生成并可视化完全高保真的3D/4D器官几何模型。传统的3D/4D医学图像重建需要近数百个投影，这不仅耗费了难以承受的计算时间，而且会给人体带来不必要的高成像/辐射剂量。此外，后续分割或提取准确的3D器官模型总是需要进一步的过程。减少投影次数可以减少计算时间和成像剂量，但重建图像质量也会相应下降。据我们所知，目前还没有从一幅二维医学灰度图像中直接、显式地重建多个三维器官网格的方法。对于单视图2D医学图像，例如3D/4D-CT投影或X射线图像，我们的端到端DeepOrganNet框架可以通过基于三变量张量积变形技术的多个模板学习平滑变形场，利用从输入2D图像提取的信息潜在描述符，高效和有效地重建具有各种几何形状的3D/4D肺部模型。该方法可以保证为3D/4D肺部模型生成高质量和高保真的流形网格，而目前所有基于深度学习的单图像形状重建方法都不能保证生成高质量的流形网格。这项工作的主要贡献是从2D单视投影准确地重建3D器官形状，显著缩短操作时间以实现即时可视化，并显著减少对人类受试者的成像剂量。通过使用大量的3D和4D示例，包括合成体模和真实患者数据集，对实验结果进行了评估，并与传统的重建方法和深度学习的最新进展进行了比较。结果表明，该方法仅需几毫秒即可生成10K个顶点的器官网格，在实时图像引导放射治疗(IGRT)中具有很大的应用潜力。
This paper introduces a deep neural network based method, i.e., DeepOrganNet, to generate and visualize fully high-fidelity 3D / 4D organ geometric models from single-view medical images with complicated background in real time. Traditional 3D / 4D medical image reconstruction requires near hundreds of projections, which cost insufferable computational time and deliver undesirable high imaging / radiation dose to human subjects. Moreover, it always needs further notorious processes to segment or extract the accurate 3D organ models subsequently. The computational time and imaging dose can be reduced by decreasing the number of projections, but the reconstructed image quality is degraded accordingly. To our knowledge, there is no method directly and explicitly reconstructing multiple 3D organ meshes from a single 2D medical grayscale image on the fly. Given single-view 2D medical images, e.g., 3D / 4D-CT projections or X-ray images, our end-to-end DeepOrganNet framework can efficiently and effectively reconstruct 3D / 4D lung models with a variety of geometric shapes by learning the smooth deformation fields from multiple templates based on a trivariate tensor-product deformation technique, leveraging an informative latent descriptor extracted from input 2D images. The proposed method can guarantee to generate high-quality and high-fidelity manifold meshes for 3D / 4D lung models; while, all current deep learning based approaches on the shape reconstruction from a single image cannot. The major contributions of this work are to accurately reconstruct the 3D organ shapes from 2D single-view projection, significantly improve the procedure time to allow on-the-fly visualization, and dramatically reduce the imaging dose for human subjects. Experimental results are evaluated and compared with the traditional reconstruction method and the state-of-the-art in deep learning, by using extensive 3D and 4D examples, including both synthetic phantom and real patient datasets. The efficiency of the proposed method shows that it only needs several milliseconds to generate organ meshes with 10K vertices, which has great potential to be used in real-time image guided radiation therapy (IGRT).