CAREER: Advancing Adversarial Robustness of Natural Language Generation Systems

商业、法律、医疗保健和军方的决策者使用自然语言处理系统从海量数据中获取见解，并做出更明智的决策。最近，自然语言生成系统(NLGS)变得流行起来。例如，用于促进公共健康的问答系统和聊天机器人，以及用于应急和预防犯罪的社会传感系统。然而，攻击者可能能够操纵这些系统，从而导致糟糕的输出和糟糕的决策，这是有风险的。深度学习系统对攻击者的健壮性已经成为机器学习和安全领域的一个活跃话题，而基于NLG的系统的健壮性研究相对较少。解决这一问题很重要，因为深度学习和NLG系统使用的数据和算法的性质以及它们所用于的任务类型有许多不同。该项目将通过全面研究自然语言生成系统容易受到的攻击类型来解决这些差异，开发自然语言生成系统漏洞的数学模型，并通过改变NLG系统的设计来减少这些漏洞的策略。这反过来将导致更安全、更值得信赖的NLG系统，并为参与研究和相关课程的学生提供许多教育机会。该项目的总体目标是了解NLG系统的攻击面和漏洞，并开发新的经验和理论方法来增强其对抗能力。这项工作将以NLG的两项常见任务--摘要和问答--为基础，并围绕三个相互关联的目标展开。第一个是开发一个框架，提出一种新的基于人工智能的优化方法，用于测试NLG系统对各种攻击模型的攻击。第二是对导致此类攻击的漏洞进行深入分析和表征。第三是开发一套防御方法和工具，以增强NLG系统的健壮性。这项研究将与教育和推广相结合，为妇女和代表性不足的群体提供研究经验，将研究成果纳入课程内容开发和课程设计，并组织研讨会和比赛以缩小NLP和网络安全计划之间的差距。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Decision-makers in business, legal, healthcare, and the military use natural language processing systems to obtain insights from vast amounts of data and to make more informed decisions. Recently, natural language generation systems (NLGs) are becoming popular. Examples include question and answer systems and chatbots that are used for advancing public health, and social sensing systems that are used for emergency response and crime prevention. However, there are risks that attackers may be able to manipulate these systems leading to poor outputs and poor decision-making. Robustness to adversaries in deep learning systems has become an active topic in the machine learning and security communities, but the robustness of NLG-based systems is much less studied. This is important to address because there are many differences in the nature of the data and algorithms deep learning and NLG systems employ, as well as the types of tasks they are used for. This project will address these differences through a comprehensive look at the kinds of attacks natural language generation systems are vulnerable to, developing both mathematical models of their vulnerabilities and strategies for reducing them through changes in how NLG systems are designed. This, in turn, will lead to safer, more trustworthy NLG systems and provide a number of educational opportunities for students involved in the research and related classes.The overall goal of the project is to understand NLG systems' attack surface and vulnerabilities and develop novel empirical and theoretical methods for increasing their adversarial robustness. The work will be grounded in two common NLG tasks, summarization and question-answering, and structured around three interconnected aims. The first is developing a framework and proposing novel AI-based optimization methods for examining NLG systems against various attack models. The second is having an in-depth analysis and characterization of vulnerabilities that lead to such attacks. The third is developing a set of defensive methods and tools for enhancing the robustness of NLG systems. This research will be integrated with education and outreach by providing research experiences for women and underrepresented groups, incorporating research results into the course content development and curriculum design, and organizing workshops and competitions to reduce the gap between NLP and cybersecurity programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.