Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration.
Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration.
复制标题
DOI：
10.1007/978-3-030-59716-0_32
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Unberath M
中科院分区：
文献类型：
--
作者：
Gao C;Liu X;Gu W;Killeen B;Armand M;Taylor R;Unberath M
微分渲染是一种将3D场景与相应的2D图像连接起来的技术。由于它是可微的，因此可以学习图像形成期间的过程。以前的可区分渲染方法集中在3D场景的基于网格的表示上，这对于其中使用体积、体素化模型来表示解剖结构的医学应用是不合适的。我们提出了一种新的投影空间Transformer模块，将空间变换器推广到投影几何，从而实现可微分体绘制。我们证明了这种架构的有用性的例子，2D/3D射线照片和CT扫描之间的注册。具体来说，我们证明了我们的Transformer能够实现图像处理和投影模型的端到端学习，该模型近似于相对于姿势参数为凸的图像相似性函数，因此可以使用传统的梯度下降进行有效优化。据我们所知，我们是第一个描述的投影透射成像的背景下，包括渲染和姿态估计的空间变换器。我们希望我们的发展将有利于相关的3D研究应用。源代码可在https://github.com/gaocong13/Projective-Spatial-Transformers上获得。
Differentiable rendering is a technique to connect 3D scenes with corresponding 2D images. Since it is differentiable, processes during image formation can be learned. Previous approaches to differentiable rendering focus on mesh-based representations of 3D scenes, which is inappropriate for medical applications where volumetric, voxelized models are used to represent anatomy. We propose a novel Projective Spatial Transformer module that generalizes spatial transformers to projective geometry, thus enabling differentiable volume rendering. We demonstrate the usefulness of this architecture on the example of 2D/3D registration between radiographs and CT scans. Specifically, we show that our transformer enables end-to-end learning of an image processing and projection model that approximates an image similarity function that is convex with respect to the pose parameters, and can thus be optimized effectively using conventional gradient descent. To the best of our knowledge, we are the first to describe the spatial transformers in the context of projective transmission imaging, including rendering and pose estimation. We hope that our developments will benefit related 3D research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.