SaTC: CORE: Small: Systematic Threat Characterization and Prevention in Open-Domain Dialog Systems

由深度神经网络驱动的对话系统或聊天机器人越来越多地被大规模部署，而不了解影响它们的漏洞。使用专门设计的学习算法，这些聊天机器人被训练成从现有的人与人之间的对话数据中学习，以产生关于各种主题的令人信服的对话。然而，训练数据中的偏差，包括故意注入的偏差，可能会使这些系统成熟，容易被恶意行为者滥用，目的是引发有毒或有害的对话。这可能会使易受攻击的用户面临潜在的危害，因为现有的部署缺乏对安全的关注，而且它们被用于医疗保健、情感支持和美国司法系统等敏感领域。该项目将系统地描述影响聊天机器人系统的各种威胁，然后构建新的可部署防御系统，以测量毒性、发现隐藏的漏洞、对受影响的系统进行解毒，并启用攻击弹性训练管道。该项目还将在多个计算机科学学科之间以及工业界和学术界之间建立伙伴关系，以提高对这些威胁的认识并加以防御。该项目为代表不足的K-12学生提供了独特的机会，学习机器学习和安全领域的新兴主题，旨在吸引他们进入STEM职业生涯。这个项目有三个研究推动力。首先是使用广泛使用的聊天机器人管道进行大规模测量研究，以表征它们对无意和故意注入的毒性的脆弱性。毒性注入攻击的特点是使用一种新颖的、全自动的管道，该管道利用大型语言模型，只需最少的人工监督，允许方法规模扩展。第二个推动力是开发一种新的生成性建模方法，以探测聊天机器人的隐藏毒性漏洞，并对模型进行解毒并创建安全基准。第三个推力建立在早期发现的基础上，以开发一种新的攻击不可知性培训管道，该管道对毒物注射攻击具有弹性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Dialog systems or chatbots powered by deep neural networks are increasingly being deployed at scale without understanding the vulnerabilities impacting them. Using specially designed learning algorithms, these chatbots are trained to learn from existing human-human conversation data to produce convincing conversations on a variety of topics. However, biases in the training data, including intentionally injected ones, can make these systems ripe for abuse by malicious actors who aim to trigger toxic or harmful conversations. This may expose vulnerable users to potential harms, given the lack of attention to security in existing deployments and the fact that they are used in sensitive domains such as healthcare, emotional support, and the U.S. justice system. This project will systematically characterize a variety of threats impacting chatbot systems, then build novel deployable defenses to measure toxicity, uncover hidden vulnerabilities, detoxify impacted systems, and enable attack-resilient training pipelines. The project will also create partnerships between multiple computer science disciplines and between industry and academia to raise awareness of and defend against these threats. The project provides unique opportunities to underrepresented K-12 students to study emerging topics in the field of machine learning and security, aiming to attract them towards STEM careers. This project has three research thrusts. The first is conducting a large-scale measurement study using widely used chatbot pipelines to characterize their vulnerability to unintentionally and intentionally injected toxicity. Toxicity injection attacks are characterized using a novel, fully automated pipeline that leverages large language models with minimal human supervision, allowing the methods to scale. The second thrust is developing a novel generative modeling approach to probe chatbots for hidden toxicity vulnerabilities, and to detoxify models and create safety benchmarks. The third thrust builds on the earlier findings to develop a novel attack-agnostic training pipeline that is resilient to toxicity injection attacks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.