On-the-fly data synthesis for deep learning-based analysis of 3D+t microscopy experiments

多维荧光显微镜可以以高空间和时间分辨率捕获整个模型生物的3D视频(3D+t)。自动图像分析可以用来检测和分割像细胞核和质膜这样的荧光标记结构，并在TB级的3D+T图像数据集中跟踪它们的时间动态。然而，到目前为止，还没有可用的算法来自动提供无错误的分割和跟踪结果。此外，基于深度学习的方法可能非常适合于这类分析任务，但由于严重缺乏合适的训练数据、有限的GPU内存以及人工标注所需的极大时间工作量，基于深度学习的方法还不能充分适用于这些大规模图像分析问题。生成的数据将能够通过为大规模3D+T问题提供基准数据集，来训练渴求数据的有监督的深度学习模型，并扩展现有的图像分析竞赛。作为第一步，我们将创建常用模型生物的真实地面图像，包括模拟胚胎表面形状和胚胎内人造细胞的真实时空分布。模拟的基本事实将包括细胞核和质膜的分割掩膜、真实的细胞帧到帧位移、细胞周期阶段以及应对细胞分裂事件的完整细胞谱系。基于生成性对抗网络，我们将开发图像生成器，使用不成对的图像到图像转换和实例级条件，从标签图像域生成逼真的、时间分辨率的3D显微图像。这涉及到新的拓扑保持损失的发展和向模型提供关于全局图像上下文的辅助信息的新方法，使得对小的局部斑块的处理将仍然坚持样本的全局模式。所有开发的构建块将作为通用软件框架实施，该框架适用于即时生成具有可参数化难度水平的数据。这将使课程学习变得流畅，而不必手动注释单个原始图像。例如，监督分割模型的训练可以从相对清晰的图像开始，并且可以通过降低信噪比、对象外观甚至对象运动和细胞周期阶段来依次提高难度。由于可以提供给模型的训练数据实际上是无限的，而且真实感达到了一个新的水平，我们预计所开发的方法将在很大程度上解决现有训练数据不足的问题，并为基于3D+T显微镜的实验获得良好的泛化深度神经网络。

Multidimensional fluorescence microscopy allows capturing 3D videos (3D+t) of entire model organisms at high spatial and temporal resolution. Automated image analysis can be used to detect and segment fluorescently labeled structures like cell nuclei and plasma membranes and to follow their temporal dynamics in terabyte-scale 3D+t image data sets. However, to date there are no algorithms available yet that provide error-free segmentation and tracking results automatically. Moreover, deep learning-based methods that are potentially perfectly suited for such analysis tasks are not sufficiently applicable to these large-scale image analysis problems yet, due to a significant lack of suitable training data, limited GPU memory and the extreme time effort required for manual annotations.The aim of the proposed project is to develop new methods for generating synthetic training data that can be used for 3D segmentation and tracking tasks in developmental biology. Generated data will enable the training of data-hungry supervised deep learning models and for extending existing image analysis competitions by providing benchmark data sets for large-scale 3D+t problems. As a first step, we will create realistic ground truth images of frequently used model organisms including simulation of the embryo surface shapes and a realistic spatiotemporal distribution of artificial cells within the embryos. Simulated ground truth will encompass segmentation masks of cell nuclei and plasma membranes, realistic frame-to-frame displacements of cells, cell cycle stages as well as the complete cell lineage to cope with cell division events. Based on generative adversarial networks, we will develop image generators that produce realistically looking, time-resolved 3D microscopy images from the label image domain using unpaired image-to-image translation and instance-level conditioning. This involves the development of new topology-preserving losses and new ways of supplying the models with auxiliary information about the global image context such that processing of small local patches will still adhere to the global patterns of the specimen. All developed building blocks will be implemented as a generic software framework that is suitable for on-the-fly data generation with parametrizable difficulty levels. This will enable smooth curriculum learning without having to annotate a single raw image manually. For instance, training of a supervised segmentation model could start with relatively crisp images and the difficulty level can be successively raised by decreasing the signal to noise ratio, the object appearance or even object movements and cell cycle stages. Due to an effectively unlimited amount of training data that can be fed to the model and a new level of realism, we envision that the developed methods will largely solve the lack of available training data and to obtain well-generalizing deep neural networks for 3D+t microscopy-based experiments.