CAREER: Semantic Multi-Task Learning for Generalizable and Interpretable Language Generation

自然语言生成（NLG）系统在我们周围有几个重要的应用，例如，将长文档自动总结和简化为简短有用的摘要的任务，或者自动描述周围视觉信息流以帮助视力残疾人的视频字幕的任务，或者预测对话中的下一个响应的对话系统。当前最先进的NLG系统擅长生成在单词和短语（语法）级别上正确的“浅”输出。然而，他们缺乏几个重要的语义“知识技能”，这一项目解决：（1）避免输出信息是矛盾的或无关的给定输入，（2）能够提取最重要的信息主题从大型输入文档或视频，（3）保持一个正确的顺序顺序的句子和段落。此外，该项目将侧重于使这些自动化系统更具可解释性，即，使他们能够向人类解释他们的决定，这使他们在与学生和残疾人接触时更安全，更值得信赖。由此产生的知识增强型NLG系统将在他们以前从未见过的新场景中更加强大。这将允许技术广泛访问和社会影响力，通过允许值得信赖的，有吸引力的代理，可以为各种真实世界的应用程序生成更自然和准确的语言，例如视觉-语音障碍的自动助理，医疗保健和学校中自动个人助理的智能辅导，以及机器人-人类协作（例如，用于导航、组装和故障排除的口头说明）。该项目提供了如何增强NLG模型的关键语言语义知识技能的技术，例如，逻辑蕴涵，以避免矛盾和不相关的信息输入，显着性，以提取最重要的信息子集，和话语结构，以加强连贯的秩序，在生成的文本。这将通过一个通用的多任务学习（MTL）框架来实现，该框架通过共享参数和模型组件联合训练手头的主要NLG模型和辅助技能模型（蕴涵，显着性和话语）。重点1将研究如何共享特定的模型组件（例如，更高的任务不可知层与更低的任务相关层）可以通过领域不可知的知识转移导致更强和更普遍的任务性能。Thrust 2将开发自我学习的多任务学习模型，可以避免昂贵的手动调整，并通过多臂强盗和基于强化奖励的控制器自动决定哪些最佳辅助技能任务与主要任务共享（以及哪些模型组件）。推力3将贡献新的控制器奖励，允许域可转移性，而不会损害域内任务性能。最后，这些模型在解释其生成语言中的语义错误时也将更具可解释性，以及可视化自学习MTL模型所做的共享决策。该项目将全面评估知识增强的NLG模型在几个不同的NLG任务的文档摘要，数据到文档，和视频字幕。这项工作还将包括发布一套公共的辅助知识技能和MTL框架，以促进其他NLG任务的概括性和可解释性的进步。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Natural language generation (NLG) systems has several important applications around us, e.g., the task of automatically summarizing and simplifying long documents into a short useful summary, or the task of video captioning to automatically describe a stream of surrounding visual information for assisting persons with visual disability, or a dialogue system that predicts the next response in a conversation. Current state-of-the-art NLG systems are good at generating 'shallow' outputs which are correct at the word and phrase (syntax) level. However, they lack several important semantic "knowledge skills", which this project addresses: (1) avoiding output information that is contradictory or unrelated to the given input, (2) being able to extract the most important topics of information from the large input document or video, and (3) maintaining a correctly-ordered sequence of sentences and paragraphs. Moreover, the project will focus on making these automated systems more interpretable, i.e., enable them to explain their decisions to humans, which makes them safer and more trustworthy when interfacing with students and persons with disability. The resulting knowledge-enhanced NLG systems will be more robust in new unseen scenarios that they have not seen before. This will allow making the technology widely accessible and societally impactful, by allowing trustworthy, engaging agents that can generate more natural and accurate language for diverse, real-world applications such as automated assistants for vision-speech impairments, intelligent tutoring by automated personal assistants in healthcare and schools, as well as for robot-human collaboration (e.g., verbal instructions for navigation, assembly, and troubleshooting). This project contributes techniques on how to enhance NLG models with crucial linguistic-semantic knowledge skills e.g., logical entailment to avoid contradictory and unrelated information with respect to the input, saliency to extract the most important information subsets, and discourse structure to enforce coherent order in the generated text. This will be achieved via a general multi-task learning (MTL) framework, which jointly trains the primary NLG model at hand with the auxiliary skill models (of entailment, saliency, and discourse) via shared parameters and model components. Thrust 1 will study how sharing specific model components (e.g., higher task-agnostic versus lower task-dependent layers) via flexible sharing strengths can lead to stronger and generalized task performance via domain-agnostic knowledge transfer. Thrust 2 will develop self-learned multi-task learning models that can avoid expensive manual tuning and automatically decide what the best auxiliary skill tasks are to share with the primary task (and which model components), via multi-armed bandit and reinforcement reward-based controllers. Thrust 3 will contribute novel controller rewards that allow domain-transferability without hurting in-domain task performance. Finally, these models will also be more interpretable in explaining their semantic errors in the generated language, as well as in visualizing what sharing decisions the self-learning MTL model made. The project will comprehensively evaluate the knowledge-enhanced NLG models on several diverse NLG tasks of document summarization, data-to-document, and video captioning. The effort will also include the release of a public suite of the auxiliary knowledge skills and MTL framework for promoting generalization and interpretability advancements in other NLG tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.