Neural networks for joint modelling of medical images and radiology reports

神经网络在医学图像分析方面取得了显著的成果，具有促进医疗保健的潜力。应用这些模型的常见任务包括x射线、CT和MRI中的病理分类和分割。纯映像模型有各种限制。例如，它们只能从为图像绘制的手动注释中学习，例如绘制病理轮廓，这是有限的，因为制作它们昂贵，耗时且需要临床专业知识。从这些有限的数据中学习导致模型泛化欠佳。此外，模型不能解释做出预测的原因，因为它们没有与人类用户“沟通”的机制。在临床实践中，放射科医生用自然语言编写书面报告，描述医学图像的内容。这样的报告可以描述存在的病理，其位置，大致大小，外观等。这些报告每天在每个放射科大量生成，并包含相应图像和病理的丰富信息。然而，当前一代的机器学习模型无法充分利用这些丰富的数据。目的和目的本项目的目的是开发能够共同处理和学习医学图像和相应的书面放射学报告的神经网络。具体来说，我们将开发能够：a)处理输入图像并自动生成描述其内容的文本的学习框架和模型。这种自动生成的报告对于加速放射学工作流程是非常宝贵的。b)使用书面报告中的信息来学习如何检测图像中的病理。来自大量放射学报告的信息将与来自较小的人工注释图像数据库（例如绘制的病理轮廓）的信息相结合，以训练比当前一代模型更有效的疾病检测模型，这些模型只能从有限的人工注释中学习。c)联合建模图像和语言将使模型能够解释其预测。这样的模型将能够使用自然语言向人类描述为什么要对疾病分割做出特定的预测，反之亦然。该项目属于EPSRC医学成像研究领域和医疗保健技术主题。拟议的研究将产生新的技术，用于联合建模医学数据库中的视觉和文本信息，这是当前一代算法仍然不足的领域。这样的模型将能够从常规生成的放射学报告中提取丰富的信息，以学习如何更好地检测医学扫描中的病理。当这些模型在新的扫描中对病理的存在做出预测时，这些模型也能够用自然语言为他们的预测生成解释，这些解释可以很容易地被人类用户解释。这些机制将提高模型的预测性能和可解释性，这些因素是在临床工作流程中可靠采用此类工具所必需的。

Neural networks have achieved remarkable results for medical image analysis with potential to facilitate healthcare. Examples of common tasks where these models are applied include classification and segmentation of pathology in X-rays, CT and MRI. Image-only models have various limitations. For example, they can only learn from manual annotations made for an image, such the drawn outline of a pathology, which are limited because making them is expensive, time-consuming and requires clinical expertise. Learning from this limited data leads to models with suboptimal generalization. Moreover, models cannot explain the reason for making a prediction, as they have no mechanism of "communicating" with the human user.In clinical practice, a radiologist produces a written report in natural language, describing contents of a medical image. Such a report can describe what pathology exists, its location, approximate size, appearance, and more. These reports are produced daily in large quantities in every Radiology department and hold rich information about the corresponding images and pathologies therein. Current generation of machine learning models, however, cannot exploit this wealth of data adequately. Aim and ObjectivesAim of this project is to develop neural networks that can jointly process and learn from medical images and corresponding written radiology reports. Specifically, we will develop learning frameworks and models that are able to:a) Process an input image and automatically generate text that describes its contents. This automatically generated report can be invaluable for accelerating radiology workflows.b) Use information from written reports to learn how to detect a pathology in an image. The information from the numerous radiology reports will be combined with information from a smaller database of manually annotated images (e.g. drawn outlines of pathologies) to train disease detection models that are more potent than the current generation of models, which can only learn from the limited manual annotations.c) Jointly modelling images and language will give models the capability to explain their predictions. Such models will be able to use natural language to describe to a human why a specific prediction is made about disease segmentation, and vice versa.Importance and Contributions to KnowledgeThis project falls within the EPSRC Medical Imaging research area and the Healthcare Technologies theme.The proposed research will generate novel techniques for jointly modelling visual and textual information from medical databases, an area where current generation of algorithms is still inadequate. Such models will be able to extract wealth of information from routinely produced radiology reports to learn how to better detect pathologies in medical scans. When these models make a prediction for the existence of a pathology in a new scan, these models will also be able to generate explanations for their prediction in natural language that can be easily interpreted by the human user. These mechanisms will enhance predictive performance and explainability of the models, factors that are necessary for reliable adoption of such tools in clinical workflows.