Upsampling of low-resolution/large-volume 3D tomographic images using generative adversarial neural networks applied to biological anthropology, medical imaging, and evolutionary biology

“重要的是要记住，信息不是知识，知识不是智慧，智慧也不是远见。但信息是通向所有这些的第一步，”亚瑟·C·克拉克爵士说。在所有依赖成像的研究领域，大容量和高分辨率信息是一个固有的难题；你可以做到其中之一，也可以做到另一个，但永远不能两者兼得。在保持大背景的同时提高分辨率的动机是多方面的，从对基本生物现象的全面分析，到维持成像中的辐射安全标准，再到保护具有文化价值的稀有标本的完整性。然而，随着神经网络深度学习的应用，外推不存在的空间信息一直是一个不适定的问题，因为它的解决方案不是唯一的。虽然已有大量文献介绍了使用超分辨率卷积神经网络(CNN)、递归神经网络(RNN)或生成对抗网络(GAN)对单幅图像进行提升的方法，但大多数算法都是针对合成下采样的2D图像设计的(并在此基础上进行了验证)。在这里，我们融合了深度学习、断层X射线成像、生物矿化、生物人类学和牙齿放射学方面的专业知识，使用历史(古代木乃伊)、临床(牙锥束计算机断层扫描)和基础科学(鸟类蛋壳)样本的3D图像设计并验证了开放式上采样算法。我们项目的独特之处在于提供了原始的多尺度3D图像集，这些图像集跨越多个分辨率/体积尺度，可以在3D中准确地叠加(配准)，并且可以专业地分割(具有在体素基础上识别和分配给目标类别的有意义的特征)。图像是分层的，这意味着局部特征的邻域与特征本身一样重要；实际上，这是CNN操作的基础，该操作基于特征的上下文(以及上下文的上下文)来识别和标记特征。GAN包括构造人工特征的生成器算法，以及比较人工特征和真实特征的鉴别器算法：两者的迭代导致收敛，并构建逼真的人工空间信息。通过将GaN-CNN应用于低分辨率/大体积图像，以高分辨率/小体积图像作为地面事实，我们将实现3D图像的上采样×n。为了避免数据量(×n3)的不可避免地增加，我们将实现一种减少体素深度的并行分割算法，因为最终的目标是上采样图像的分割。最后，我们将使用在多倍放大下获得的古代木乃伊、鸟类蛋壳和人类颅面复合体的注册系列3D图像来探索3D图像上采样的限制。这项开放式研究将把生物人类学、动物学和临床放射学带入生物成像和3D图像分析的新层次。

'It is vital to remember that information is not knowledge, that knowledge is not wisdom, and that wisdom is not foresight. But information is the first essential step to all of these', Sir Arthur C. Clarke.Large-volume versus high-resolution information is an inherent conundrum in all domains of research that rely on imaging; you can achieve either one, or the other, but never both. The incentives to increase resolution while keeping large context are many, from comprehensive analysis of basic biological phenomena, to maintaining radiation safety standards in imaging, to preserving the integrity of rare specimens of cultural value. However, extrapolating nonexisting spatial information has always been an ill-posed problem because its solution is not unique ' until now with the application of deep learning using neural networks. Although there is ample literature on upscaling single images using super-resolution convolutional neural networks (CNN), recurrent neural networks (RNN) or generative adversarial networks (GAN), most algorithms are designed for (and validated on) synthetically downsampled 2D images. Here we merge expertise in deep learning, tomographic X-ray imaging, biomineralization, biological anthropology, and dental radiology to design and validate an open-ended upsampling algorithm using 3D images of historical (ancient mummies), clinical (dental cone-beam computed tomography) and basic science (bird eggshell) samples. Unique to our project is the availability of original multi-scale 3D image sets that span multiple resolution/volume scales and which can be accurately superimposed (registered) in 3D, and can be expertly segmented (having meaningful features identified and assigned to the target class on a voxel basis). Images are hierarchical, meaning that the neighborhood of local features is as important as the features themselves; indeed, that is the basis of the CNN operation which identifies and labels features based on their context (and the context of context). GANs include a generator algorithm that constructs artificial features, and a discriminator algorithm that compares artificial and true features: iteration of the two leads to convergence, and to construction of realistic artificial spatial information. By applying GAN-CNN to a low-resolution/large-volume image using a high-resolution/low-volume image as ground truth, we will achieve 3D image upsampling ×n. To circumvent the inevitable increase of the data size (×n3) we will implement a parallel segmentation algorithm that reduces voxel depth, because the ultimate objective is the segmentation of upsampled images. Finally, we will explore the limit of image upsampling in 3D using registered series 3D images of ancient mummies, bird eggshells and human craniofacial complex, acquired at multiple magnifications. This open-ended study will bring biological anthropology, zoology and clinical radiology towards the new tier in bioimaging and 3D image analysis.