When Sparsity Meets Contrastive Models: Less Graph Data Can Bring Better Class-Balanced Representations
When Sparsity Meets Contrastive Models: Less Graph Data Can Bring Better Class-Balanced Representations
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
2.9
通讯作者：
Chunhui Zhang;Chao Huang;Yijun Tian;Qianlong Wen;Z. Ouyang;Youhuan Li;Yanfang Ye;Chuxu Zhang-Chuxu-Zha
中科院分区：
文献类型：
--
作者：
Chunhui Zhang;Chao Huang;Yijun Tian;Qianlong Wen;Z. Ouyang;Youhuan Li;Yanfang Ye;Chuxu Zhang-Chuxu-Zha
图神经网络（GNN）是一种针对非欧数据的强大模型，但其训练过程中往往会产生大量不必要的计算：一方面，非欧数据的密度不规则，导致其计算代价较高;另一方面，通常与非欧几里德数据相关联的类不平衡属性不能通过数据的连续性来减轻，从而阻碍了模型的推广。为了解决上述问题，从理论上讲，我们首先假设使用GNN训练数据子集的有效性，这是由子集和全集之间的梯度距离保证的。从经验上讲，我们还观察到，数据的一个子集可以为模型优化提供信息梯度，并且随着时间的推移动态变化。我们把这种现象称为动态数据稀疏。此外，我们发现修剪稀疏对比模型可能会“错过”有价值的信息，导致信息子集上的大损失值。受上述发现的启发，我们开发了一个名为Data Dec antation（DataDec）的统一数据模型动态稀疏性框架来解决上述挑战。DataDec的核心思想是通过稀疏图对比学习在训练过程中动态识别信息子集。在图基准数据集上对DataDec的有效性进行了全面评估，并在图像数据上验证了其泛化能力。
Graph Neural Networks (GNNs) are powerful models for non-Euclidean data, but their training is often accentuated by massive unnecessary computation: On the one hand, training on non-Euclidean data has relatively high computational cost due to its irregular density properties; on the other hand, the class imbalance property often associated with non-Euclidean data cannot be alleviated by the massiveness of the data, thus hindering the generalisation of the models. To address the above issues, theoretically, we start with a hypothesis about the effectiveness of using a subset of training data for GNNs, which is guaranteed by the gradient distance between the subset and the full set. Empirically, we also observe that a subset of the data can provide informative gradients for model optimization and which changes over time dynamically. We name this phenomenon dynamic data sparsity. Additionally, we find that pruned sparse contrastive models may “miss” valuable information, leading to a large loss value on the informative subset. Motivated by the above findings, we develop a unified data model dynamic sparsity framework called Data Dec antation (DataDec) to address the above challenges. The key idea of DataDec is to identify the informative subset dynamically during the training process by applying sparse graph contrastive learning. The effectiveness of DataDec is comprehensively evaluated on graph benchmark datasets and we also verify its generalizability on image data.