SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation
SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation
复制标题
DOI：
10.48550/arxiv.2210.12213
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Zekun Li;Jina Kim;Yao-Yi Chiang;Muhao Chen
中科院分区：
文献类型：
--
作者：
Zekun Li;Jina Kim;Yao-Yi Chiang;Muhao Chen
命名地理实体（简称地理实体）是许多地理数据集的构建块。描述地理实体是各种应用领域的组成部分，如地理智能和地图理解，而一个关键的挑战是捕捉实体的空间变化的背景。我们假设，我们应该知道一个地理实体的特点，其周围的实体，类似于知道单词的意义，他们的语言背景。因此，我们提出了一种新的空间语言模型，SpaBERT，它提供了一个通用的地理实体表示的基础上相邻的地理空间数据的实体。SpaBERT扩展了BERT来捕获线性化的空间上下文，同时结合了空间坐标嵌入机制来保留二维空间中实体的空间关系。SpaBERT使用掩码语言建模和掩码实体预测任务进行预训练，以学习空间依赖关系。我们将SpaBERT应用于两个下游任务：地理实体分类和地理实体链接。与现有的语言模型，不使用空间上下文相比，SpaBERT表现出显着的性能改善这两个任务。我们还分析了SpaBERT在各种设置下的实体表示以及空间坐标嵌入的效果。
Named geographic entities (geo-entities for short) are the building blocks of many geographic datasets. Characterizing geo-entities is integral to various application domains, such as geo-intelligence and map comprehension, while a key challenge is to capture the spatial-varying context of an entity. We hypothesize that we shall know the characteristics of a geo-entity by its surrounding entities, similar to knowing word meanings by their linguistic context. Accordingly, we propose a novel spatial language model, SpaBERT, which provides a general-purpose geo-entity representation based on neighboring entities in geospatial data. SpaBERT extends BERT to capture linearized spatial context, while incorporating a spatial coordinate embedding mechanism to preserve spatial relations of entities in the 2-dimensional space. SpaBERT is pretrained with masked language modeling and masked entity prediction tasks to learn spatial dependencies. We apply SpaBERT to two downstream tasks: geo-entity typing and geo-entity linking. Compared with the existing language models that do not use spatial context, SpaBERT shows significant performance improvement on both tasks. We also analyze the entity representation from SpaBERT in various settings and the effect of spatial coordinate embedding.