CLIP-Lite: Information Efficient Visual Representation Learning with Language Supervision-猫眼课题宝

DOI：

发表时间：

2021-12

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

A. Shrivastava;Ramprasaath R. Selvaraju;N. Naik;Vicente Ordonez

我们提出了CLIP-Lite，一种信息高效的方法，通过与文本注释的特征对齐进行视觉表示学习。与之前提出的CLIP模型相比，CLIP-Lite在优化其对比学习目标时，对于每个阳性图文样本，只需要一个阴性图文样本对。我们通过利用一个信息有效的下限，以最大限度地提高两个输入模态之间的互信息来实现这一点。这使得CLIP-Lite能够以显著减少的数据量和批量大小进行训练，同时在相同规模下获得比CLIP更好的性能。我们通过在COCO-Captions数据集上进行预训练并测试迁移学习到其他数据集来评估CLIP-Lite。CLIP-Lite在Pascal VOC分类上获得了+14.0%mAP的绝对性能增益，在ImageNet上获得了+22.1%top-1的准确率增益，同时与其他更复杂的文本监督模型相当或上级。CLIP-Lite在图像和文本检索、零镜头分类和视觉基础方面也上级CLIP。最后，我们表明CLIP-Lite可以利用语言语义来鼓励无偏见的视觉表示，可用于下游任务。实施：https://github.com/4m4n5/CLIP-Lite

We propose CLIP-Lite, an information efficient method for visual representation learning by feature alignment with textual annotations. Compared to the previously proposed CLIP model, CLIP-Lite requires only one negative image-text sample pair for every positive image-text sample during the optimization of its contrastive learning objective. We accomplish this by taking advantage of an information efficient lower-bound to maximize the mutual information between the two input modalities. This allows CLIP-Lite to be trained with significantly reduced amounts of data and batch sizes while obtaining better performance than CLIP at the same scale. We evaluate CLIP-Lite by pretraining on the COCO-Captions dataset and testing transfer learning to other datasets. CLIP-Lite obtains a +14.0% mAP absolute gain in performance on Pascal VOC classification, and a +22.1% top-1 accuracy gain on ImageNet, while being comparable or superior to other, more complex, text-supervised models. CLIP-Lite is also superior to CLIP on image and text retrieval, zero-shot classification, and visual grounding. Finally, we show that CLIP-Lite can leverage language semantics to encourage bias-free visual representations that can be used in downstream tasks. Implementation: https://github.com/4m4n5/CLIP-Lite

CLIP-Lite: Information Efficient Visual Representation Learning with Language Supervision

文献摘要

相似文献