CAREER: Advancing Adversarial Robustness of Natural Language Generation Systems

商业、法律的、医疗保健和军事领域的决策者使用自然语言处理系统从大量数据中获得见解，并做出更明智的决策。近年来，自然语言生成系统（NLGs）越来越受欢迎。例子包括用于促进公共卫生的问答系统和聊天机器人，以及用于应急响应和预防犯罪的社会传感系统。然而，攻击者可能能够操纵这些系统，从而导致不良的输出和不良的决策。深度学习系统对对手的鲁棒性已经成为机器学习和安全社区的一个活跃话题，但基于NLG的系统的鲁棒性研究得很少。这一点很重要，因为深度学习和NLG系统所使用的数据和算法的性质以及它们所用于的任务类型存在许多差异。该项目将通过全面研究自然语言生成系统易受攻击的类型来解决这些差异，开发其漏洞的数学模型以及通过改变NLG系统设计方式来减少漏洞的策略。这反过来将导致更安全，更值得信赖的NLG系统，并为参与研究和相关课程的学生提供大量的教育机会。该项目的总体目标是了解NLG系统的攻击面和漏洞，并开发新的经验和理论方法，以提高其对抗鲁棒性。这项工作将以两个常见的NLG任务为基础，总结和问答，并围绕三个相互关联的目标进行结构化。第一个是开发一个框架，并提出新的基于AI的优化方法，用于检查NLG系统对各种攻击模型的影响。第二是对导致此类攻击的漏洞进行深入分析和描述。第三是开发一套增强NLG系统鲁棒性的防御方法和工具。这项研究将通过为妇女和代表性不足的群体提供研究经验，将研究结果纳入课程内容的制定和课程设计，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的评估被认为值得支持。影响审查标准。

Decision-makers in business, legal, healthcare, and the military use natural language processing systems to obtain insights from vast amounts of data and to make more informed decisions. Recently, natural language generation systems (NLGs) are becoming popular. Examples include question and answer systems and chatbots that are used for advancing public health, and social sensing systems that are used for emergency response and crime prevention. However, there are risks that attackers may be able to manipulate these systems leading to poor outputs and poor decision-making. Robustness to adversaries in deep learning systems has become an active topic in the machine learning and security communities, but the robustness of NLG-based systems is much less studied. This is important to address because there are many differences in the nature of the data and algorithms deep learning and NLG systems employ, as well as the types of tasks they are used for. This project will address these differences through a comprehensive look at the kinds of attacks natural language generation systems are vulnerable to, developing both mathematical models of their vulnerabilities and strategies for reducing them through changes in how NLG systems are designed. This, in turn, will lead to safer, more trustworthy NLG systems and provide a number of educational opportunities for students involved in the research and related classes.The overall goal of the project is to understand NLG systems' attack surface and vulnerabilities and develop novel empirical and theoretical methods for increasing their adversarial robustness. The work will be grounded in two common NLG tasks, summarization and question-answering, and structured around three interconnected aims. The first is developing a framework and proposing novel AI-based optimization methods for examining NLG systems against various attack models. The second is having an in-depth analysis and characterization of vulnerabilities that lead to such attacks. The third is developing a set of defensive methods and tools for enhancing the robustness of NLG systems. This research will be integrated with education and outreach by providing research experiences for women and underrepresented groups, incorporating research results into the course content development and curriculum design, and organizing workshops and competitions to reduce the gap between NLP and cybersecurity programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.