TopoBERT: Exploring the topology of fine-tuned word representations
TopoBERT: Exploring the topology of fine-tuned word representations
复制标题
DOI：
10.1177/14738716231168671
复制
发表时间：
2023-05
影响因子：
2.3
通讯作者：
Archit Rathore;Yichu Zhou;Vivek Srikumar;Bei Wang
中科院分区：
文献类型：
--
作者：
Archit Rathore;Yichu Zhou;Vivek Srikumar;Bei Wang
基于变压器的语言模型及其变体在自然语言处理中发现了宽度的使用（NLP）。尚不清楚微调过程如何影响这些模型嵌入单词的基础结构，我们提出了一种视觉分析系统，用于互动地探索各种微调过程基于变压器的模型 - 从拓扑角度使用多个微调批处理更新，模型的后续层和不同的NLP任务。 topobert的模型嵌入空间。模型对，（2）可视化多个尺度和层的嵌入空间的形状，（3）将有关输入数据集的语言和上下文信息与嵌入式空间的拓扑结合在一起。举例说明其在探索微型单词嵌入的应用程序。这些见解。
Transformer-based language models such as BERT and its variants have found widespread use in natural language processing (NLP). A common way of using these models is to fine-tune them to improve their performance on a specific task. However, it is currently unclear how the fine-tuning process affects the underlying structure of the word embeddings from these models. We present TopoBERT, a visual analytics system for interactively exploring the fine-tuning process of various transformer-based models – across multiple fine-tuning batch updates, subsequent layers of the model, and different NLP tasks – from a topological perspective. The system uses the mapper algorithm from topological data analysis (TDA) to generate a graph that approximates the shape of a model’s embedding space for an input dataset. TopoBERT enables its users (e.g. experts in NLP and linguistics) to (1) interactively explore the fine-tuning process across different model-task pairs, (2) visualize the shape of embedding spaces at multiple scales and layers, and (3) connect linguistic and contextual information about the input dataset with the topology of the embedding space. Using TopoBERT, we provide various use cases to exemplify its applications in exploring fine-tuned word embeddings. We further demonstrate the utility of TopoBERT, which enables users to generate insights about the fine-tuning process and provides support for empirical validation of these insights.