Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures
Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures
复制标题
DOI：
10.1109/sp46214.2022.9833572
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Eugene Bagdasaryan;Vitaly Shmatikov
中科院分区：
文献类型：
--
作者：
Eugene Bagdasaryan;Vitaly Shmatikov
我们研究了神经序列到序列（seq2seq）模型的一个新威胁：训练时间攻击，导致模型“旋转”其输出，以支持对手选择的情感或观点，但只有当输入包含对手选择的触发词时。例如，一个spinned 1摘要模型输出任何提到某个人或组织名称的文本的正面摘要。传统的后门会导致模型在触发器的输入上产生不正确的输出，而旋转模型的输出会保留上下文并保持标准的准确性度量，但也会满足对手选择的元任务。模型旋转使宣传成为服务，其中宣传被定义为有偏见的言论。攻击者可以创建定制的语言模型，为所选触发器生成所需的旋转，然后部署这些模型以生成虚假信息（平台攻击），或者将其注入ML训练管道（供应链攻击），将恶意功能转移到受害者训练的下游模型。为了证明模型旋转的可行性，我们开发了一种新的后门技术。它堆叠一个对抗性元任务（例如，情感分析）到Seq2Seq模型上，反向传播期望的元任务输出（例如，积极情绪）到词嵌入空间中的点，我们称之为“伪词”，并使用伪词来移动seq2seq模型的整个输出分布。我们评估这种攻击的语言生成，摘要和翻译模型与不同的触发器和元任务，如情绪，毒性和蕴涵。自旋模型在很大程度上保持其准确性指标（ROUGE和BLEU），同时改变其输出以满足对手的元任务。我们还表明，在供应链攻击的情况下，自旋功能转移到下游模型。最后，我们提出了一个黑盒子，元任务独立的防御，给定一个候选触发器列表，可以检测模型，选择性地应用自旋输入与任何这些制造商。1我们使用“spinned”，而不是“spinned”，以匹配如何在公共关系中使用这个词。
We investigate a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to “spin” their outputs so as to support an adversary-chosen sentiment or point of view—but only when the input contains adversary-chosen trigger words. For example, a spinned 1 summarization model outputs positive summaries of any text that mentions the name of some individual or organization.Model spinning introduces a “meta-backdoor” into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary.Model spinning enables propaganda-as-a-service, where propaganda is defined as biased speech. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy these models to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models trained by victims.To demonstrate the feasibility of model spinning, we develop a new backdooring technique. It stacks an adversarial meta-task (e.g., sentiment analysis) onto a seq2seq model, backpropagates the desired meta-task output (e.g., positive sentiment) to points in the word-embedding space we call “pseudo-words,” and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models largely maintain their accuracy metrics (ROUGE and BLEU) while shifting their outputs to satisfy the adversary’s meta-task. We also show that, in the case of a supply-chain attack, the spin functionality transfers to downstream models.Finally, we propose a black-box, meta-task-independent defense that, given a list of candidate triggers, can detect models that selectively apply spin to inputs with any of these triggers.1We use “spinned” rather than “spun” to match how the word is used in public relations.