Answering questions about medical images

视觉问答（VQA）[1]是回答关于图像内容的自由文本问题的任务。例如，给定一个房间的图像-“角落里的椅子是什么颜色的？”回答这个问题需要对视觉、语言、空间推理和常识的共同理解。得益于深度学习的进步，计算机视觉领域的技术水平在过去几年中一直在稳步提高，语言模型最近经历了快速发展，而空间推理和常识仍然是最难以捉摸的。医学成像为VQA研究提供了大量有趣的特定领域挑战[2]。例如，图像可以是三维扫描，专家词汇可以用于问题和答案（如空间关系的解剖方向术语），但另一方面，我们有诸如UMLS的本体，可以支持医学常识知识。该项目将专注于计算机视觉和自然语言处理的一系列方法论方法，它们共同支持回答有关2D和3D放射学图像的问题。此外，VQA的可解释性将在这里得到解决-它既是一个引人注目的基础研究方向，也是医疗保健应用的理想考虑因素。

Visual Question Answering (VQA) [1] is the task of answering free-text questions about the content of images. For example, given an image of a room - 'What colour is the chair standing in the corner?' - answering this question requires joint understanding of vision, language, spatial reasoning, and common sense. Thanks to advances in deep learning the state of the art in computer vision has been improving steadily over the past few years and language models experienced rapid progress recently, while spatial reasoning and common sense remain the most elusive.Medical imaging offers plenty of interesting domain-specific challenges for VQA research [2]. For example, the images can be 3-dimensional scans, specialist vocabulary can be used in questions and answers (such as anatomical directional terms for spatial relations), but on the other hand, we have ontologies such as UMLS which can support medical common sense knowledge.This project will focus on a range of methodological approaches in computer vision and natural language processing, which collectively support answering questions about 2D and 3D radiology images.Also, the interpretability of VQA will be addressed here - it is both a compelling basic research direction, and a desired consideration for healthcare applications.