Learning NLP Tasks with Multimodal Interactions

该项目旨在开发一种方法，使用户能够使用自然语言指令、对话和其他形式的反馈（如标签）组合来改变机器学习模型的行为。其目的是让用户纠正错误，提供缺失的信息，并使模型适应新的任务和领域。具体来说，该项目侧重于多标签文本分类的深度神经网络。该方法将结合上下文嵌入和贝叶斯方法的最新进展，主动请求用户输入并处理解释不同形式反馈的不确定性。该项目将积极寻找真实世界的用例，并结合用户研究来激励交互式学习方法的设计和评估。核心目标是开发一种方法，允许用户使用多种反馈来源（包括自然语言输入）来适应和纠正深度神经网络中的错误。该项目旨在开展交互式机器学习的基础研究。该项目目前正在与主管的现有合作伙伴Yvette Payne博士合作进行其中一个确定的案例研究。本案例研究特别涉及自动注释GP转录本的过程。本研究的总体目的是创建一个系统，当提供咨询录音时，该系统可以自动创建和注释一套简明的咨询笔记，我们的意图是通过开发一个系统来扩展此功能，该系统可以提供实时诊断以及建议的进一步查询线。我们的目的是创造一种基于主动学习概念的新方法。然而，当主动学习向用户查询额外的标记数据时，我们的方法将向用户查询额外的信息。以案例研究为例，该系统将向全科医生提供一个问题，让病人提出一个问题，从而为建议的诊断产生较低的不确定性评分。目前的目的是使用包含转录全科医生咨询的数据集来训练系统。[1]虽然该方法最初是在医疗保健和全科医生咨询领域开发的，但它可以很容易地应用于专业人员和不熟悉该领域的人之间进行咨询的任何环境。所需要做的就是改变一下培训材料。通过在确定的案例研究中成功开发这种方法以及任何必要的伦理审查，可能有可能在NHS中部署这样一个系统，允许所有全科医生咨询笔记的标准化。该项目属于EPSRC信息和通信技术（ICT）和医疗保健技术主题。除此之外，它还属于人工智能和机器人技术这一更广泛的研究领域杰普森，M.，索尔兹伯里，C.，里德，M.，梅特卡夫，C.，加塞德，L.和巴恩斯，R., 2017。“百万分之一”研究：创建英国初级保健咨询数据库。英国医学杂志，67(658),pp.e345-e351。

This project aims to develop a methodology that enables users to alter the behaviour of machine learning models using a combination of natural language instructions, dialogue, and other forms of feedback such as labels. The aim is for users to correct errors, provide missing information, and adapt models to new tasks and domains. Specifically, the project focuses on deep neural networks for multi-label text classification. The methodology will combine recent advances in contextualised embeddings with Bayesian approaches to actively request user inputs and handle the uncertainty in interpreting different forms of feedback. The project will actively seek out real-world use cases and incorporate user studies to motivate the design and evaluation of interactive learning methods.The core objective is to develop a methodology that allows users to adapt and correct errors in deep neural networks using multiple sources of feedback, including natural language inputs. This project aims to carry out foundational research into interactive machine learning.The project is currently undertaking one of the identified case-studies in collaboration with one of the supervisor's existing partners - Dr. Yvette Payne. This case study specifically involves the process of automatically annotating GP transcripts. With the overall intention of the study being the creation of a system that can automatically create and annotate a concise set of consultation notes when provided an audio recording of the consultation, it is our intention to expand beyond this by developing a system that can provide real-time diagnoses along with suggested further lines of inquiry. It is our intention to create a novel methodology based off the concept of active learning. However, where active learning queries the user for additional labelled data, our methodology will instead query the user for additional information. Using the case study as an example, the system will provide the GP with a question to ask the patient that will yield a lower uncertainty score for the suggested diagnosis. The current intention is to train the system using a dataset containing transcribed GP consultations. [1]While the methodology is initially being developed within the domain of healthcare and GP consultations, it can easily be applied to any context where a consultation is taking place between a professional and someone unfamiliar with that domain. All that requires is a change in the training material.Through the successful development of this methodology within the identified case study along with any required ethical reviews, it may be possible to deploy such a system for use within the NHS, allowing for the standardisation of all GP consultation notes.This project falls within the EPSRC Information and Communication Technologies (ICT), and Healthcare Technologies themes. As well as this, it falls under the wider research area of Artificial Intelligence and Robotics.[1] Jepson, M., Salisbury, C., Ridd, M., Metcalfe, C., Garside, L. and Barnes, R., 2017. The 'One in a Million' study: creating a database of UK primary care consultations. British Journal of General Practice, 67(658), pp.e345-e351.