Learning-based multi-modal registration and data fusion for robust cancer tissue classification

该提案旨在开发基于无监督学习和卷积神经网络（CNN）的新型多模态图像配准方法。与针对每对新图像进行优化的经典方法相反，CNN学习一个配准函数，该函数通过高度并行化的神经网络计算单次通过的最佳变换，在GPU上花费几分之一秒或在CPU上花费几秒钟（Fu et al. 2019）。当考虑到在3D研究中必须对齐数十到数百个切片时，这种加速尤为重要。该项目可大致分为以下三个工作包：1。伪影感知3D组织学重建的无监督配准首先要解决的挑战是模态内配准，例如将连续组织学切片对齐到重建的3D体积和随后的可视化中。我们将开发基于空间Transformer网络的方法来执行连续组织切片的顺序注册，以及CNN来检测切片过程中生成的图像中的异常，例如组织撕裂，折叠，压缩，不一致的染色和丢失组织。这些信息将被集成到配准工作流程中，以表示高变形性区域。2.多模态图像配准的表示学习将开发神经网络架构，用于从不同类型的数据（组织学，MSI，MRI）中进行无监督表示学习，优化不同成像格式之间实现准确多模态配准（基于已知变换）的功能。了解数据中的哪些特征是共同的可以提高对哪些方面可以在模态之间进行可靠比较的理解，并导致开发更简单的模型和配准工作流程。还将探讨开发一种基于模式独立学习的配准工作流程的可能性。这将联合收割机结合上述创新的成果，旨在为工作流程提供来自同一样本的组织学、MSI和MRI数据，并获得一组将每张图像转换为通用3D空间的转换。3.我们将评估所提出的方法与经典迭代方法（elastix，niftyreg，ANTS）以及最近基于学习的配准框架（如VoxelMorph）的性能比较（Balakrishnan et al. 2018）。我们设想，所提出的方法将吸引生物医学研究人员的兴趣，在临床前或手术后的癌症研究中，由于研究设计的细微差别，经常采用特设的方法。该注册方法还将被集成到现有的多模态图像分析开源框架（https：//github.com/AlanRace/SpectralAnalysis）中，并向科学界发布。这将进一步增加所开发方法的影响，使其成为免费提供和易于使用的软件，减少进入的障碍

This proposal aims to develop novel multi-modal image registration methods based on unsupervised learning and convolutional neural networks (CNNs). In contrast to classical methods that optimise for each new pair of images, CNNs learn a registration function that computes the optimal transformation from single pass through a heavily parallelised neural network, taking a fraction of a second on a GPU or a few seconds on a CPU (Fu et al. 2019). This speed-up is especially important when considering the tens to hundreds of sections that must be aligned in 3D studies. The project may be broadly organised into the following three work packages:1. Unsupervised registration for artefact-aware 3D histology reconstructionThe first challenge to address is intra-modality registration, such as the alignment of serial histology sections, into a reconstructed 3D volume and subsequent visualisation. We will develop methods based on spatial transformer networks toperform sequential registration of serial histological sections, and CNNs to detect anomalies in the images generated through the sectioning process, such as tissue tearing, folding, compression, inconsistent staining and missing tissue. Thisinformation will be integrated into the registration workflow, to signify regions of high deformability.2. Representation learning for multi-modal image registrationNeural network architectures will be developed for unsupervised representation learning from different styles of data (histology, MSI, MRI), optimising for features which enable accurate multi-modal registration (based on known transformations) between the different imaging formats. Understanding which features in the data are common may improve the understanding of which aspects can be reliably compared between modalities as well as leading to the development of simpler models and registration workflows. The possibility of developing a modality independent learning based registration workflow will also be explored. This will combine the outcomes of the innovations described above,with the aim of providing histology, MSI and MRI data from the same sample to the workflow and obtaining a set of transformations converting each image into a common 3D space.3. Comparison with state-of-the-art and releaseWe will evaluate the performance of the proposed method in comparison to classical iterative methods (elastix, niftyreg, ANTS), as well as recent learning-based registration frameworks such as VoxelMorph (Balakrishnan et al. 2018). We envisage that the proposed methods will attract interest among biomedical researchers working on pre-clinical or postsurgical cancer studies, where ad hoc methods are frequently adopted due to the nuances of the study design. The registration method will also be integrated into an existing open-source framework for multi-modal image analysis (https://github.com/AlanRace/SpectralAnalysis) and released to the scientific community. This will further increase the impactof the developed methods, by making them available in freely available and easy-to-use software, reducing the barrier to entry