On-the-fly data synthesis for deep learning-based analysis of 3D+t microscopy experiments

多维荧光显微镜可以捕获整个模型生物体的3D视频（3D+t）在高空间和时间分辨率。自动图像分析可用于检测和分割荧光标记的结构，如细胞核和质膜，并在TB级3D+t图像数据集中跟踪其时间动态。然而，到目前为止，还没有算法可以自动提供无错误的分割和跟踪结果。此外，基于深度学习的方法可能非常适合这种分析任务，但由于严重缺乏合适的训练数据，有限的GPU内存和手动注释所需的极端时间工作。拟议项目的目的是开发新的方法来生成可用于3D分割的合成训练数据，发育生物学中的追踪任务生成的数据将能够训练数据饥渴的监督式深度学习模型，并通过为大规模3D+t问题提供基准数据集来扩展现有的图像分析竞赛。作为第一步，我们将创建经常使用的模型生物的真实地面图像，包括胚胎表面形状的模拟和胚胎内人工细胞的真实时空分布。模拟的地面真相将包括细胞核和质膜的分割掩模，细胞的真实帧到帧位移，细胞周期阶段以及完整的细胞谱系，以科普细胞分裂事件。基于生成对抗网络，我们将开发图像生成器，使用不成对的图像到图像转换和实例级条件反射，从标签图像域生成逼真的时间分辨3D显微图像。这涉及到发展新的拓扑保持损失和新的方式提供的模型与辅助信息的全球图像的背景下，这样的处理小的局部补丁仍将坚持全球模式的标本。所有开发的构件将作为一个通用的软件框架，适用于在飞行中的数据生成参数化的难度水平。这将使顺利的课程学习，而不必手动注释一个单一的原始图像。例如，监督分割模型的训练可以从相对清晰的图像开始，并且可以通过降低信噪比、对象外观或甚至对象运动和细胞周期阶段来连续提高难度水平。由于可以提供给模型的训练数据量实际上是无限的，并且具有新的现实主义水平，我们设想所开发的方法将在很大程度上解决缺乏可用训练数据的问题，并为基于3D+t显微镜的实验获得良好的泛化深度神经网络。

Multidimensional fluorescence microscopy allows capturing 3D videos (3D+t) of entire model organisms at high spatial and temporal resolution. Automated image analysis can be used to detect and segment fluorescently labeled structures like cell nuclei and plasma membranes and to follow their temporal dynamics in terabyte-scale 3D+t image data sets. However, to date there are no algorithms available yet that provide error-free segmentation and tracking results automatically. Moreover, deep learning-based methods that are potentially perfectly suited for such analysis tasks are not sufficiently applicable to these large-scale image analysis problems yet, due to a significant lack of suitable training data, limited GPU memory and the extreme time effort required for manual annotations.The aim of the proposed project is to develop new methods for generating synthetic training data that can be used for 3D segmentation and tracking tasks in developmental biology. Generated data will enable the training of data-hungry supervised deep learning models and for extending existing image analysis competitions by providing benchmark data sets for large-scale 3D+t problems. As a first step, we will create realistic ground truth images of frequently used model organisms including simulation of the embryo surface shapes and a realistic spatiotemporal distribution of artificial cells within the embryos. Simulated ground truth will encompass segmentation masks of cell nuclei and plasma membranes, realistic frame-to-frame displacements of cells, cell cycle stages as well as the complete cell lineage to cope with cell division events. Based on generative adversarial networks, we will develop image generators that produce realistically looking, time-resolved 3D microscopy images from the label image domain using unpaired image-to-image translation and instance-level conditioning. This involves the development of new topology-preserving losses and new ways of supplying the models with auxiliary information about the global image context such that processing of small local patches will still adhere to the global patterns of the specimen. All developed building blocks will be implemented as a generic software framework that is suitable for on-the-fly data generation with parametrizable difficulty levels. This will enable smooth curriculum learning without having to annotate a single raw image manually. For instance, training of a supervised segmentation model could start with relatively crisp images and the difficulty level can be successively raised by decreasing the signal to noise ratio, the object appearance or even object movements and cell cycle stages. Due to an effectively unlimited amount of training data that can be fed to the model and a new level of realism, we envision that the developed methods will largely solve the lack of available training data and to obtain well-generalizing deep neural networks for 3D+t microscopy-based experiments.