Real-time Image Deblurring to Improve Throughput of Serial-Section Volume Electron Microscopy for Neural Connectomic Studies
Real-time Image Deblurring to Improve Throughput of Serial-Section Volume Electron Microscopy for Neural Connectomic Studies
复制标题
实时图像去模糊可提高神经连接组学研究中串行切片体积电子显微镜的吞吐量
DOI：
10.1093/micmic/ozad067.494
复制
发表时间：
2023
影响因子：
2.8
通讯作者：
Wei, D
中科院分区：
文献类型：
--
作者：
Schalek, R L;Parikh, N;Lichtman, J W;Wei, D
生成大的连续切片电子显微镜体积需要自动化和可靠的图像采集。特别是，图像质量可靠性至关重要，因为重拍失焦或软聚焦图像会大大降低整体成像吞吐量。例如，在收集由~ 5000个连续切片组成的1.3 PB数据集时，从一个切片获取图像需要~ 30分钟，并且包含45,750张图像（总计约2.3亿张图像）[1]。重新拍摄2%的图像需要每天至少一次打破自动化和人为干预。由于大多数低质量图像显示软焦点，使用机器学习的后图像校正可能是图像重新获取的替代方案。失焦或模糊图像的产生有几个原因：聚焦算法的问题，污名算法的问题，表面缺陷或非平坦的视场。消除图像重新采集工作不仅提高了大型串行EM数据集的吞吐量，而且还提高了较小数据集的整体图像采集性能，其中人工成本、存储成本和机器租赁成本是主要的项目考虑因素。为了提高串行切片EM的吞吐成像率，我们开发了一种基于机器学习的实时图像去模糊工具，消除了重新提取失焦图像的需要。我们使用Zhang等人[2]提出的方法对失焦图像进行去模糊处理。该方法基于图推理注意网络（GRAN），而不是基于更传统的深度卷积神经去模糊网络。该方法首先进行特征降尺度，然后将特征馈送到图推理注意块（GRAB）中，然后进行上尺度，从而提高了计算效率和更大的接受区域。我们将提取的特征点建模为视觉组件，并构造一个全连通关系图。接下来，使用图卷积网络（GCN）分析关系图。GCN结合残差学习给出了我们的GRAB块。从GRAB获得的特征可以作为注意力，然后放大并用于生成校正后的图像。该模型通过计算逐像素L1损失和地面真实图像与校正图像之间的对抗损失来训练。图1所示的SEM图像比较了失焦图像与ML校正后的图像以及正确聚焦的图像。数据表明，对于大多数失焦图像，校正后的图像与地面真值图像非常相似。PNSR的变化表明，与失焦图像相比，校正后的图像有了显著的改善。正如预期的那样，严重失焦的图像无法正确校正。去模糊算法的有效性极限目前正在测试中。将通过在图像采集管道中包括校正算法来实例化去模糊算法的实现。这里使用内部算法[3]确定图像质量。如果图像质量低于预定的阈值，失焦图像将使用去模糊算法进行校正
Generating large serial-section electron microscopy volumes requires automated and reliable image acquisition. In particular, image quality reliability is of paramount importance as retaking out-of-focus, or soft-focused images dramatically decreases the overall imaging throughput. As an example, when collecting a 1.3 PB data set consisting of∼ 5000 serial sections, acquiring images from one section took∼ 30 min and consisted of 45,750 images (∼ 230 million total images)[1]. Retaking 2% of these images requires breaking the automation and human intervention at least once per day. Since most low-quality images exhibit soft focus, post image correction with machine learning, may be an alternative to image re-acquisition. Out-of-focus, or blurred images arise for several reasons: a problem with the focusing algorithm, a problem with the stigmation algorithm, surface defects, or non-flat fields-of-view. Removal of image re-acquisition efforts not only improves large serial EM dataset throughput, but also improves the overall image acquisition performance of smaller datasets where labor costs, storage costs, and machine rental costs are major project considerations. To increase the throughput imaging rate of serial section EM, we have developed a machine learningbased, real-time image deblurring tool that eliminates the need for retaking out-of-focus images. We use a method proposed by Zhang et al.[2] to deblur out-of-focus images. This method is based on a graph reasoning attention network (GRAN) and not on more traditional deep convolutional neural deblurring networks. The method initially performs feature downscaling before feeding the features to a graph reasoning attention block (GRAB) and then performs upscaling, resulting in efficient computation and a larger receptive region. We model the extracted feature points as visual components and construct a fully connected relationship graph. Next, a graph convolutional network (GCN) analyses the relationship graph. The GCN combined with the residual learning gives us our GRAB blocks. The features obtained from GRAB can be treated as attention, then upscaled and used to generate the corrected image. The model is trained by computing the pixel-wise L1 loss and an adversarial loss between the ground truth image and the corrected image. The SEM images presented in Figure 1 compares out-of-focus image with the ML corrected image, and a properly focused image. The data indicate that for most out-of-focus images the corrected images are very similar to the ground truth images. The change in PNSR indicates a significant improvement of the corrected images compared to the out-of-focus images. As expected, severely out-of-focus images cannot be properly corrected. The limits of effectiveness of the deblurring algorithm is currently being tested.Implementation of the deblurring algorithm will be instantiated by including the correction algorithm in the image acquisition pipeline. Here the image quality is determined using an in-house algorithm [3]. If the image quality is below a predetermined threshold, the out-of-focus image will be corrected using the deblurring algorithm.[4]