GMML is All You Need
GMML is All You Need
复制标题
DOI：
10.1109/icip49359.2023.10222150
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Sara Atito;Muhammad Awais;J. Kittler
中科院分区：
文献类型：
--
作者：
Sara Atito;Muhammad Awais;J. Kittler
视觉转换器(VITS)在计算机视觉领域引起了极大的兴趣，因为它们在利用上下文信息方面具有灵活性，无论是严格受限的局部信息还是远程全局信息。然而，众所周知，他们需要数据，因此经常在大规模数据集上进行预训练，例如JFT-300M或ImageNet。无论数据集的大小如何，理想的学习方法都会表现得最好，这是当前学习方法所缺乏的一个特性，只有几个现有的工作用有限的数据研究VITS。我们提出了组掩蔽模型学习(GMML)，这是一种自监督学习(SSL)方法，它能够在有限数据预训练的情况下训练VITS并获得最先进的性能(SOTA)。GMML使用图像中所有概念所传达的信息。这是通过操作随机连接的令牌组，连续覆盖图像内容的不同有意义部分，然后从概念的可见部分恢复隐藏的信息来实现的。与现有的大多数SSL方法不同，GMML不需要动量编码器，也不依赖于大批量和渐变停止等仔细的实现细节。有关预培训、优化和评估代码的信息，请访问：https://github.com/GMML.
Vision transformers (ViTs) have generated significant interest in the computer vision community because of their flexibility in exploiting contextual information, whether it is sharply confined local, or long range global. However, they are known to be data hungry and therefore often pretrained on large-scale datasets, e.g. JFT-300M or ImageNet. An ideal learning method would perform best regardless of the size of the dataset, a property lacked by current learning methods, with merely a few existing works studying ViTs with limited data. We propose Group Masked Model Learning (GMML), a self-supervised learning (SSL) method that is able to train ViTs and achieve state-of-the-art (SOTA) performance when pre-trained with limited data. The GMML uses the information conveyed by all concepts in the image. This is achieved by manipulating randomly groups of connected tokens, successively covering different meaningful parts of the image content, and then recovering the hidden information from the visible part of the concept. Unlike most of the existing SSL approaches, GMML does not require momentum encoder, nor relies on careful implementation details such as large batches and gradient stopping. Pretraining, finetuning, and evaluation codes are available under: https://github.com/GMML.