Fast Certification of Vision-Language Models Using Incremental Randomized Smoothing
Fast Certification of Vision-Language Models Using Incremental Randomized Smoothing
复制标题
DOI：
10.1109/satml59370.2024.00019
复制
发表时间：
2023-11
期刊：
影响因子：
--
通讯作者：
Ashutosh Nirala;Ameya Joshi;Chinmay Hegde;Soumik Sarkar
中科院分区：
文献类型：
--
作者：
Ashutosh Nirala;Ameya Joshi;Chinmay Hegde;Soumik Sarkar
诸如剪辑之类的深视觉模型的一个关键好处是，它们可以启用零拍开的词汇分类。用户可以在推理时通过自然语言提示来定义新颖的类标签。但是，尽管基于夹的零击分类器已经在一系列域转移中表现出竞争性能，但它们仍然很容易受到对抗性攻击的影响。因此，确保此类模型的鲁棒性对于它们在野外的可靠部署至关重要。在这项工作中，我们引入了开放式词汇认证（OVC），这是一种为开放式录音带模型而设计的快速认证方法，例如通过随机平滑技术（例如夹子）。鉴于基本的“培训”提示及其相应的认证剪辑分类器，OVC依赖于这样的观察，即可以将带有新颖提示的分类器视为基础训练集中附近分类器的扰动版本。因此，OVC可以使用增量随机平滑的变化快速证明新型分类器。通过使用缓存技巧，我们在新提示的认证过程中达到了大约两个数量级加速度。为了实现进一步的（启发式）加速，OVC使用多元正态分布在给定输入处近似于嵌入空间，从而绕过通过视觉主链进行采样的需求。我们使用CIFAR-10和ImageNet测试数据集上的多个视觉语言主机通过实验评估来证明OVC在实验评估中的有效性。
A key benefit of deep vision-language models such as CLIP is that they enable zero-shot open vocabulary classification; the user has the ability to define novel class labels via natural language prompts at inference time. However, while CLIP-based zero-shot classifiers have demonstrated competitive performance across a range of domain shifts, they remain highly vulnerable to adversarial attacks. Therefore, ensuring the robustness of such models is crucial for their reliable deployment in the wild.In this work, we introduce Open Vocabulary Certification (OVC), a fast certification method designed for open-vocabulary models like CLIP via randomized smoothing techniques. Given a base "training" set of prompts and their corresponding certified CLIP classifiers, OVC relies on the observation that a classifier with a novel prompt can be viewed as a perturbed version of nearby classifiers in the base training set. Therefore, OVC can rapidly certify the novel classifier using a variation of incremental randomized smoothing. By using a caching trick, we achieve approximately two orders of magnitude acceleration in the certification process for novel prompts. To achieve further (heuristic) speedups, OVC approximates the embedding space at a given input using a multivariate normal distribution bypassing the need for sampling via forward passes through the vision backbone. We demonstrate the effectiveness of OVC on through experimental evaluation using multiple vision-language backbones on the CIFAR-10 and ImageNet test datasets.