Answering questions about medical images

点击翻译按钮获取中文摘要

Visual Question Answering (VQA) [1] is the task of answering free-text questions about the content of images. For example, given an image of a room - 'What colour is the chair standing in the corner?' - answering this question requires joint understanding of vision, language, spatial reasoning, and common sense. Thanks to advances in deep learning the state of the art in computer vision has been improving steadily over the past few years and language models experienced rapid progress recently, while spatial reasoning and common sense remain the most elusive.Medical imaging offers plenty of interesting domain-specific challenges for VQA research [2]. For example, the images can be 3-dimensional scans, specialist vocabulary can be used in questions and answers (such as anatomical directional terms for spatial relations), but on the other hand, we have ontologies such as UMLS which can support medical common sense knowledge.This project will focus on a range of methodological approaches in computer vision and natural language processing, which collectively support answering questions about 2D and 3D radiology images.Also, the interpretability of VQA will be addressed here - it is both a compelling basic research direction, and a desired consideration for healthcare applications.