Ablating Concepts in Text-to-Image Diffusion Models
Ablating Concepts in Text-to-Image Diffusion Models
复制标题
DOI：
10.1109/iccv51070.2023.02074
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Nupur Kumari;Bin Zhang;Sheng-Yu Wang;Eli Shechtman;Richard Zhang;Jun-Yan Zhu
中科院分区：
文献类型：
--
作者：
Nupur Kumari;Bin Zhang;Sheng-Yu Wang;Eli Shechtman;Richard Zhang;Jun-Yan Zhu
大规模文本到图像的扩散模型可以生成具有强大合成能力的高保真图像。然而，这些模型通常是根据大量互联网数据进行训练的，这些数据通常包含受版权保护的材料、许可的图像和个人照片。此外，人们发现它们可以复制各种在世艺术家的风格或记住精确的训练样本。我们如何才能删除这些受版权保护的概念或图像而不从头开始重新训练模型？为了实现这一目标，我们提出了一种有效的方法来消除预训练模型中的概念，即防止目标概念的生成。我们的算法学习匹配目标样式、实例或文本提示的图像分布，我们希望将其消融为与锚概念相对应的分布。这可以防止模型在给定文本条件的情况下生成目标概念。大量的实验表明，我们的方法可以成功地防止消融概念的生成，同时保留模型中密切相关的概念。
Large-scale text-to-image diffusion models can generate high-fidelity images with powerful compositional ability. However, these models are typically trained on an enormous amount of Internet data, often containing copyrighted material, licensed images, and personal photos. Furthermore, they have been found to replicate the style of various living artists or memorize exact training samples. How can we remove such copyrighted concepts or images without retraining the model from scratch? To achieve this goal, we propose an efficient method of ablating concepts in the pretrained model, i.e., preventing the generation of a target concept. Our algorithm learns to match the image distribution for a target style, instance, or text prompt we wish to ablate to the distribution corresponding to an anchor concept. This prevents the model from generating target concepts given its text condition. Extensive experiments show that our method can successfully prevent the generation of the ablated concept while preserving closely related concepts in the model.