Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks
Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks
复制标题
DOI：
10.1145/3580305.3599376
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Bowen Jin;Yu Zhang;Qi Zhu;Jiawei Han
中科院分区：
文献类型：
--
作者：
Bowen Jin;Yu Zhang;Qi Zhu;Jiawei Han
网络上的表示学习旨在为每个节点导出有意义的向量表示，从而促进下游任务，如链接预测，节点分类和节点聚类。在异构富文本网络中，这项任务更具挑战性，因为（1）文本的存在或不存在：一些节点与丰富的文本信息相关联，而另一些则没有;（2）类型的多样性：多种类型的节点和边形成异构网络结构。由于预训练语言模型（PLM）已经证明了它们在获得广泛概括的文本表示方面的有效性，因此已经做出了大量努力将PLM纳入富文本网络上的表示学习。然而，很少有人能有效地联合考虑异构的结构（网络）信息，以及丰富的文本语义信息的每个节点。在本文中，我们提出了Heterformer，一个异构网络授权的Transformer，执行上下文文本编码和异构结构编码在一个统一的模型。具体来说，我们注入异构结构信息到每个Transformer层编码节点文本时。同时，Heterformer能够表征节点/边类型的异质性，并编码有或没有文本的节点。我们对三个任务进行了综合实验（即，链接预测，节点分类和节点聚类）对来自不同领域的三个大规模数据集进行测试，其中Heterformer显著且一致地优于竞争基线。代码可以在https://github.com/PeterGriffinJin/Heterformer上找到。
Representation learning on networks aims to derive a meaningful vector representation for each node, thereby facilitating downstream tasks such as link prediction, node classification, and node clustering. In heterogeneous text-rich networks, this task is more challenging due to (1) presence or absence of text: Some nodes are associated with rich textual information, while others are not; (2) diversity of types: Nodes and edges of multiple types form a heterogeneous network structure. As pretrained language models (PLMs) have demonstrated their effectiveness in obtaining widely generalizable text representations, a substantial amount of effort has been made to incorporate PLMs into representation learning on text-rich networks. However, few of them can jointly consider heterogeneous structure (network) information as well as rich textual semantic information of each node effectively. In this paper, we propose Heterformer, a Heterogeneous Network-Empowered Transformer that performs contextualized text encoding and heterogeneous structure encoding in a unified model. Specifically, we inject heterogeneous structure information into each Transformer layer when encoding node texts. Meanwhile, Heterformer is capable of characterizing node/edge type heterogeneity and encoding nodes with or without texts. We conduct comprehensive experiments on three tasks (i.e., link prediction, node classification, and node clustering) on three large-scale datasets from different domains, where Heterformer outperforms competitive baselines significantly and consistently. The code can be found at https://github.com/PeterGriffinJin/Heterformer.