Learning-based multi-modal registration and data fusion for robust cancer tissue classification

提出了一种基于无监督学习和卷积神经网络的多模式图像配准方法。与针对每一对新图像进行优化的经典方法不同，CNN学习配准函数，该函数通过高度并行的神经网络从单遍计算最优变换，在GPU上只需几分之一秒，在CPU上只需几秒钟(Fu等人。2019年)。当考虑到在3D研究中必须对齐的数十到数百个节段时，这种加速尤其重要。该项目可以大致组织为以下三个工作包：1.人工产物感知的3D组织重建的无监督配准首先要解决的挑战是内部配准，例如将连续的组织切片对齐成重建的3D体积和随后的可视化。我们将开发基于空间变换网络的方法来实现连续组织切片的顺序配准，并利用CNN来检测切片过程中产生的图像中的异常，如组织撕裂、折叠、压缩、不一致染色和组织缺失。这些信息将被整合到注册工作流程中，以表示高变形性区域。多模式图像配准的表示学习将开发神经网络结构，用于从不同类型的数据(组织学、MSI、MRI)进行无监督的表示学习，优化功能，从而实现不同成像格式之间的准确多模式配准(基于已知转换)。了解数据中的哪些特征是共同的，可能会提高对哪些方面可以在不同模式之间进行可靠比较的理解，并有助于开发更简单的模型和登记工作流程。还将探讨开发基于形态自主学习的注册工作流程的可能性。这将结合上述创新的成果，目的是将来自同一样本的组织学、MSI和MRI数据提供给工作流程，并获得一组将每个图像转换为共同3D空间的转换。与最先进的方法和版本的比较我们将评估所提出的方法的性能，并与经典的迭代方法(lamtix，niftyreg，ANTS)以及最近基于学习的注册框架，如VoxelMorph(Balakrishnan等人)进行比较。2018年)。我们预计，建议的方法将吸引从事临床前或手术后癌症研究的生物医学研究人员的兴趣，由于研究设计的细微差别，这些研究经常采用特别的方法。注册方法还将整合到现有的多模式图像分析(https://github.com/AlanRace/SpectralAnalysis)开源框架中，并向科学界发布。这将进一步增加已开发方法的影响，使其在免费可用和易于使用的软件中可用，从而降低进入壁垒

This proposal aims to develop novel multi-modal image registration methods based on unsupervised learning and convolutional neural networks (CNNs). In contrast to classical methods that optimise for each new pair of images, CNNs learn a registration function that computes the optimal transformation from single pass through a heavily parallelised neural network, taking a fraction of a second on a GPU or a few seconds on a CPU (Fu et al. 2019). This speed-up is especially important when considering the tens to hundreds of sections that must be aligned in 3D studies. The project may be broadly organised into the following three work packages:1. Unsupervised registration for artefact-aware 3D histology reconstructionThe first challenge to address is intra-modality registration, such as the alignment of serial histology sections, into a reconstructed 3D volume and subsequent visualisation. We will develop methods based on spatial transformer networks toperform sequential registration of serial histological sections, and CNNs to detect anomalies in the images generated through the sectioning process, such as tissue tearing, folding, compression, inconsistent staining and missing tissue. Thisinformation will be integrated into the registration workflow, to signify regions of high deformability.2. Representation learning for multi-modal image registrationNeural network architectures will be developed for unsupervised representation learning from different styles of data (histology, MSI, MRI), optimising for features which enable accurate multi-modal registration (based on known transformations) between the different imaging formats. Understanding which features in the data are common may improve the understanding of which aspects can be reliably compared between modalities as well as leading to the development of simpler models and registration workflows. The possibility of developing a modality independent learning based registration workflow will also be explored. This will combine the outcomes of the innovations described above,with the aim of providing histology, MSI and MRI data from the same sample to the workflow and obtaining a set of transformations converting each image into a common 3D space.3. Comparison with state-of-the-art and releaseWe will evaluate the performance of the proposed method in comparison to classical iterative methods (elastix, niftyreg, ANTS), as well as recent learning-based registration frameworks such as VoxelMorph (Balakrishnan et al. 2018). We envisage that the proposed methods will attract interest among biomedical researchers working on pre-clinical or postsurgical cancer studies, where ad hoc methods are frequently adopted due to the nuances of the study design. The registration method will also be integrated into an existing open-source framework for multi-modal image analysis (https://github.com/AlanRace/SpectralAnalysis) and released to the scientific community. This will further increase the impactof the developed methods, by making them available in freely available and easy-to-use software, reducing the barrier to entry