Neural networks for joint modelling of medical images and radiology reports

神经网络在医学图像分析方面取得了显著的效果，具有促进医疗保健的潜力。应用这些模型的常见任务的例子包括X射线、CT和MRI中的病理分类和分割。仅限图像的模型有各种限制。例如，他们只能从为图像做的手动注释中学习，比如绘制病理轮廓，这是有限的，因为制作它们是昂贵的、耗时的，并且需要临床专业知识。从这些有限的数据中学习会导致具有次优泛化的模型。此外，模型无法解释做出预测的原因，因为它们没有与人类用户“沟通”的机制。在临床实践中，放射科医生用自然语言生成书面报告，描述医学图像的内容。这样的报告可以描述存在什么病理，其位置，大致大小，外观，等等。这些报告每天在每个放射科大量生成，并包含有关相应图像和病理的丰富信息。然而，当前一代的机器学习模型不能充分利用这些丰富的数据。目的和目的本项目的目的是开发能够联合处理和学习医学图像和相应的书面放射学报告的神经网络。具体地说，我们将开发能够：a)处理输入图像并自动生成描述其内容的文本的学习框架和模型。这个自动生成的报告对于加速放射学工作流程非常有价值。b)使用书面报告中的信息来了解如何检测图像中的病理。来自众多放射学报告的信息将与来自较小的人工注释图像数据库(例如，绘制的病理轮廓)的信息相结合，以训练比当前一代模型更有效的疾病检测模型，后者只能从有限的手动注释中学习。这样的模型将能够使用自然语言向人类描述为什么对疾病分段做出特定的预测，反之亦然。对知识的重要性和贡献该项目属于EPSRC医学成像研究领域和医疗技术主题。拟议的研究将产生从医学数据库中联合建模视觉和文本信息的新技术，这是当前一代算法仍然不足的领域。这样的模型将能够从常规产生的放射学报告中提取丰富的信息，以学习如何在医学扫描中更好地检测病理。当这些模型在新的扫描中预测是否存在病理时，这些模型也将能够用人类用户容易解释的自然语言为其预测生成解释。这些机制将增强模型的预测性能和可解释性，这些因素是在临床工作流程中可靠采用此类工具所必需的。

Neural networks have achieved remarkable results for medical image analysis with potential to facilitate healthcare. Examples of common tasks where these models are applied include classification and segmentation of pathology in X-rays, CT and MRI. Image-only models have various limitations. For example, they can only learn from manual annotations made for an image, such the drawn outline of a pathology, which are limited because making them is expensive, time-consuming and requires clinical expertise. Learning from this limited data leads to models with suboptimal generalization. Moreover, models cannot explain the reason for making a prediction, as they have no mechanism of "communicating" with the human user.In clinical practice, a radiologist produces a written report in natural language, describing contents of a medical image. Such a report can describe what pathology exists, its location, approximate size, appearance, and more. These reports are produced daily in large quantities in every Radiology department and hold rich information about the corresponding images and pathologies therein. Current generation of machine learning models, however, cannot exploit this wealth of data adequately. Aim and ObjectivesAim of this project is to develop neural networks that can jointly process and learn from medical images and corresponding written radiology reports. Specifically, we will develop learning frameworks and models that are able to:a) Process an input image and automatically generate text that describes its contents. This automatically generated report can be invaluable for accelerating radiology workflows.b) Use information from written reports to learn how to detect a pathology in an image. The information from the numerous radiology reports will be combined with information from a smaller database of manually annotated images (e.g. drawn outlines of pathologies) to train disease detection models that are more potent than the current generation of models, which can only learn from the limited manual annotations.c) Jointly modelling images and language will give models the capability to explain their predictions. Such models will be able to use natural language to describe to a human why a specific prediction is made about disease segmentation, and vice versa.Importance and Contributions to KnowledgeThis project falls within the EPSRC Medical Imaging research area and the Healthcare Technologies theme.The proposed research will generate novel techniques for jointly modelling visual and textual information from medical databases, an area where current generation of algorithms is still inadequate. Such models will be able to extract wealth of information from routinely produced radiology reports to learn how to better detect pathologies in medical scans. When these models make a prediction for the existence of a pathology in a new scan, these models will also be able to generate explanations for their prediction in natural language that can be easily interpreted by the human user. These mechanisms will enhance predictive performance and explainability of the models, factors that are necessary for reliable adoption of such tools in clinical workflows.