Neural networks for joint modelling of medical images and radiology reports

点击翻译按钮获取中文摘要

Neural networks have achieved remarkable results for medical image analysis with potential to facilitate healthcare. Examples of common tasks where these models are applied include classification and segmentation of pathology in X-rays, CT and MRI. Image-only models have various limitations. For example, they can only learn from manual annotations made for an image, such the drawn outline of a pathology, which are limited because making them is expensive, time-consuming and requires clinical expertise. Learning from this limited data leads to models with suboptimal generalization. Moreover, models cannot explain the reason for making a prediction, as they have no mechanism of "communicating" with the human user.In clinical practice, a radiologist produces a written report in natural language, describing contents of a medical image. Such a report can describe what pathology exists, its location, approximate size, appearance, and more. These reports are produced daily in large quantities in every Radiology department and hold rich information about the corresponding images and pathologies therein. Current generation of machine learning models, however, cannot exploit this wealth of data adequately. Aim and ObjectivesAim of this project is to develop neural networks that can jointly process and learn from medical images and corresponding written radiology reports. Specifically, we will develop learning frameworks and models that are able to:a) Process an input image and automatically generate text that describes its contents. This automatically generated report can be invaluable for accelerating radiology workflows.b) Use information from written reports to learn how to detect a pathology in an image. The information from the numerous radiology reports will be combined with information from a smaller database of manually annotated images (e.g. drawn outlines of pathologies) to train disease detection models that are more potent than the current generation of models, which can only learn from the limited manual annotations.c) Jointly modelling images and language will give models the capability to explain their predictions. Such models will be able to use natural language to describe to a human why a specific prediction is made about disease segmentation, and vice versa.Importance and Contributions to KnowledgeThis project falls within the EPSRC Medical Imaging research area and the Healthcare Technologies theme.The proposed research will generate novel techniques for jointly modelling visual and textual information from medical databases, an area where current generation of algorithms is still inadequate. Such models will be able to extract wealth of information from routinely produced radiology reports to learn how to better detect pathologies in medical scans. When these models make a prediction for the existence of a pathology in a new scan, these models will also be able to generate explanations for their prediction in natural language that can be easily interpreted by the human user. These mechanisms will enhance predictive performance and explainability of the models, factors that are necessary for reliable adoption of such tools in clinical workflows.