On Separate Normalization in Self-supervised Transformers
On Separate Normalization in Self-supervised Transformers
复制标题
DOI：
10.48550/arxiv.2309.12931
复制
发表时间：
2023-09
期刊：
影响因子：
--
通讯作者：
Xiaohui Chen;Yinkai Wang;Yuanqi Du;S. Hassoun;Liping Liu
中科院分区：
文献类型：
--
作者：
Xiaohui Chen;Yinkai Wang;Yuanqi Du;S. Hassoun;Liping Liu
变压器的自监督训练方法在各个领域都表现出了卓越的性能。以前的基于transformer的模型，如masked autoencoder（MAE），通常对[CLS]符号和令牌都使用单个归一化层。在本文中，我们提出了一个简单的修改，对令牌和[CLS]符号采用单独的规范化层，以更好地捕捉它们的独特特征并提高下游任务性能。我们的方法旨在减轻对两种令牌类型使用相同的规范化统计数据的潜在负面影响，这些统计数据可能与其各自的角色没有最佳匹配。我们的经验表明，通过利用单独的归一化层，[CLS]嵌入可以更好地编码全局上下文信息，并且在其各向异性空间中分布得更均匀。当用两个单独的层替换传统的规范化层时，我们观察到图像，自然语言和图形域的平均性能提高了2.7%。
Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average 2.7% performance improvement over the image, natural language, and graph domains.