CAREER: Semantic Multi-Task Learning for Generalizable and Interpretable Language Generation

自然语言生成（NLG）系统在我们周围有几个重要的应用，例如，自动总结和简化长文档为简短有用的摘要的任务，或视频字幕的任务，以自动描述周围的视觉信息流，以帮助视力障碍的人，或预测对话中的下一个响应的对话系统。目前最先进的NLG系统擅长生成“浅层”输出，在单词和短语（语法）级别上是正确的。然而，它们缺乏几个重要的语义“知识技能”，本项目解决了这些问题：(1)避免输出与给定输入相矛盾或无关的信息，(2)能够从大型输入文档或视频中提取最重要的信息主题，以及(3)保持句子和段落的正确顺序。此外，该项目将专注于使这些自动化系统更具可解释性，即使它们能够向人类解释它们的决定，这使得它们在与学生和残疾人接触时更安全，更值得信赖。由此产生的知识增强的NLG系统将在以前从未见过的新场景中更加健壮。这将使该技术广泛使用并产生社会影响，通过允许可信赖的、引人入胜的代理，这些代理可以为各种现实世界的应用生成更自然、更准确的语言，例如视觉语言障碍的自动助手，医疗保健和学校的自动个人助理的智能辅导，以及机器人-人类协作（例如，导航、组装和故障排除的口头指令）。该项目提供了如何利用关键的语言语义知识技能来增强NLG模型的技术，例如，逻辑蕴涵以避免与输入相关的矛盾和不相关的信息，显著性以提取最重要的信息子集，以及话语结构以强制生成文本中的连贯顺序。这将通过一个通用的多任务学习（MTL）框架来实现，该框架通过共享参数和模型组件，将现有的主要NLG模型与辅助技能模型（蕴涵、显著性和话语）联合训练。Thrust 1将研究如何通过灵活的共享优势共享特定的模型组件（例如，更高的任务不可知层与更低的任务依赖层），从而通过领域不可知的知识转移带来更强和更广泛的任务性能。Thrust 2将开发自学习的多任务学习模型，该模型可以避免昂贵的手动调整，并通过多臂强盗和基于强化奖励的控制器自动决定与主要任务共享的最佳辅助技能任务（以及模型组件）。Thrust 3将提供新颖的控制器奖励，允许在不影响域内任务性能的情况下进行域转移。最后，这些模型在解释生成语言中的语义错误以及可视化自学习MTL模型所做的共享决策方面也将更具可解释性。该项目将在文档摘要、数据到文档和视频字幕等多个不同的NLG任务上全面评估知识增强的NLG模型。这项工作还将包括发布辅助知识技能和MTL框架的公共套件，以促进其他NLG任务的泛化和可解释性进步。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Natural language generation (NLG) systems has several important applications around us, e.g., the task of automatically summarizing and simplifying long documents into a short useful summary, or the task of video captioning to automatically describe a stream of surrounding visual information for assisting persons with visual disability, or a dialogue system that predicts the next response in a conversation. Current state-of-the-art NLG systems are good at generating 'shallow' outputs which are correct at the word and phrase (syntax) level. However, they lack several important semantic "knowledge skills", which this project addresses: (1) avoiding output information that is contradictory or unrelated to the given input, (2) being able to extract the most important topics of information from the large input document or video, and (3) maintaining a correctly-ordered sequence of sentences and paragraphs. Moreover, the project will focus on making these automated systems more interpretable, i.e., enable them to explain their decisions to humans, which makes them safer and more trustworthy when interfacing with students and persons with disability. The resulting knowledge-enhanced NLG systems will be more robust in new unseen scenarios that they have not seen before. This will allow making the technology widely accessible and societally impactful, by allowing trustworthy, engaging agents that can generate more natural and accurate language for diverse, real-world applications such as automated assistants for vision-speech impairments, intelligent tutoring by automated personal assistants in healthcare and schools, as well as for robot-human collaboration (e.g., verbal instructions for navigation, assembly, and troubleshooting). This project contributes techniques on how to enhance NLG models with crucial linguistic-semantic knowledge skills e.g., logical entailment to avoid contradictory and unrelated information with respect to the input, saliency to extract the most important information subsets, and discourse structure to enforce coherent order in the generated text. This will be achieved via a general multi-task learning (MTL) framework, which jointly trains the primary NLG model at hand with the auxiliary skill models (of entailment, saliency, and discourse) via shared parameters and model components. Thrust 1 will study how sharing specific model components (e.g., higher task-agnostic versus lower task-dependent layers) via flexible sharing strengths can lead to stronger and generalized task performance via domain-agnostic knowledge transfer. Thrust 2 will develop self-learned multi-task learning models that can avoid expensive manual tuning and automatically decide what the best auxiliary skill tasks are to share with the primary task (and which model components), via multi-armed bandit and reinforcement reward-based controllers. Thrust 3 will contribute novel controller rewards that allow domain-transferability without hurting in-domain task performance. Finally, these models will also be more interpretable in explaining their semantic errors in the generated language, as well as in visualizing what sharing decisions the self-learning MTL model made. The project will comprehensively evaluate the knowledge-enhanced NLG models on several diverse NLG tasks of document summarization, data-to-document, and video captioning. The effort will also include the release of a public suite of the auxiliary knowledge skills and MTL framework for promoting generalization and interpretability advancements in other NLG tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.