The effects of different levels of realism on the training of CNNs with only synthetic images for the semantic segmentation of robotic instruments in a head phantom
The effects of different levels of realism on the training of CNNs with only synthetic images for the semantic segmentation of robotic instruments in a head phantom
复制标题
不同真实度水平对仅使用合成图像训练 CNN 进行头部模型机器人仪器语义分割的影响
DOI：
10.1007/s11548-020-02185-0
复制
发表时间：
2020
影响因子：
3
通讯作者：
Mitsuishi Mamoru
中科院分区：
文献类型：
--
作者：
Heredia Perez Saul Alexis;Marques Marinho Murilo;Harada Kanako;Mitsuishi Mamoru
使用深度神经网络手动生成用于医学图像语义分割的训练数据是一项耗时且容易出错的任务。在本文中，我们研究了不同层次的真实感对机器人仪器语义分割的深度神经网络训练的影响。开发了一个交互式虚拟现实环境，用于生成机器人辅助内窥镜手术的合成图像。与早期的作品相比，我们使用物理为基础的渲染增加reality.MethodsUsing虚拟现实模拟器，复制我们的机器人设置，三个合成图像数据库与现实主义的水平不断提高生成：平面，基本的，和现实的（使用物理为基础的渲染）。这些数据库中的每一个都用于训练基于UNet的语义分割深度学习模型的20个实例。仅用合成图像训练的网络在分割160个内窥镜图像的体模上进行了评估。使用Dwass-Steel-Critchlow-Fligner非参数test.ResultsOur结果表明，现实主义的水平增加了平均交叉超过工会（mIoU）的网络上的内窥镜图像的幻影（）的网络进行了比较。平面数据集的mIoU中值为0.235，基本数据集为0.458，现实数据集为0.729。所有使用合成图像训练的网络都优于朴素分类器。此外，在消融研究中，我们表明基于物理渲染的mIoU上级仪器的纹理映射（）（0.606）背景（0.685），背景和仪器结合起来（0.672）。结论采用物理-基于渲染生成合成图像是一种有效的方法，以提高训练的神经网络的语义分割手术内窥镜图像中的器械。我们的研究结果表明，这种策略可以成为深度神经网络在语义分割任务中广泛适用的重要一步，并有助于弥合机器学习中的领域差距。
PurposeThe manual generation of training data for the semantic segmentation of medical images using deep neural networks is a time-consuming and error-prone task. In this paper, we investigate the effect of different levels of realism on the training of deep neural networks for semantic segmentation of robotic instruments. An interactive virtual-reality environment was developed to generate synthetic images for robot-aided endoscopic surgery. In contrast with earlier works, we use physically based rendering for increased realism.MethodsUsing a virtual reality simulator that replicates our robotic setup, three synthetic image databases with an increasing level of realism were generated: flat, basic, and realistic (using the physically-based rendering). Each of those databases was used to train 20 instances of a UNet-based semantic-segmentation deep-learning model. The networks trained with only synthetic images were evaluated on the segmentation of 160 endoscopic images of a phantom. The networks were compared using the Dwass–Steel–Critchlow–Fligner nonparametric test.ResultsOur results show that the levels of realism increased the mean intersection-over-union (mIoU) of the networks on endoscopic images of a phantom (). The median mIoU values were 0.235 for the flat dataset, 0.458 for the basic, and 0.729 for the realistic. All the networks trained with synthetic images outperformed naive classifiers. Moreover, in an ablation study, we show that the mIoU of physically based rendering is superior to texture mapping () of the instrument (0.606), the background (0.685), and the background and instruments combined (0.672).ConclusionsUsing physical-based rendering to generate synthetic images is an effective approach to improve the training of neural networks for the semantic segmentation of surgical instruments in endoscopic images. Our results show that this strategy can be an essential step in the broad applicability of deep neural networks in semantic segmentation tasks and help bridge the domain gap in machine learning.
登录
查看更多内容
影响因子：
2.1
作者：
Zisimopoulos O;Flouty E;Stacey M;Muscroft S;Giataganas P;Nehme J;Chow A;Stoyanov D
通讯作者：
Stoyanov D
影响因子：
4.6
作者：
van der Sluis, Pieter C.;Ruurda, Jelle P.;van Hillegersberg, Richard
通讯作者：
van Hillegersberg, Richard
DOI：
10.1109/mhs.2018.8886922
发表时间：
2018
期刊：
Micro-NanoMechatronics and Human Science
影响因子：
--
作者：
T. Masuda;K. Harada;Shun'ya Adachi;F. Arai;S. Omata;A. Morita;Taichi Kin;N. Saito;Juli Yamashita;K. Chinzei;Akiyuki Hasegawa;T. Fukuda
通讯作者：
T. Fukuda
DOI：
10.1080/03610918.2016.1146761
发表时间：
2017-01-01
影响因子：
0.9
作者：
Dolgun, Anil;Demirhan, Haydar
通讯作者：
Demirhan, Haydar
工作流程和材料标准
DOI：
10.1007/978-3-030-17490-3_5
发表时间：
2019
期刊：
Ray Tracing: A Tool for All
影响因子：
--
作者：
J. Peddie
通讯作者：
J. Peddie