IsoScore: Measuring the Uniformity of Embedding Space Utilization
IsoScore: Measuring the Uniformity of Embedding Space Utilization
复制标题
DOI：
10.18653/v1/2022.findings-acl.262
复制
发表时间：
2021-08
期刊：
影响因子：
--
通讯作者：
W. Rudman;Nate Gillman;T. Rayne;Carsten Eickhoff
中科院分区：
文献类型：
--
作者：
W. Rudman;Nate Gillman;T. Rayne;Carsten Eickhoff
最近分布式词表示的成功引起了人们对分析其空间分布特性的兴趣。一些研究表明，上下文化的词嵌入模型不能将符号各向同性地投射到向量空间中。然而，目前设计用于测量各向同性的方法，如平均随机余弦相似度和划分分数，没有经过彻底的分析，并不适合测量各向同性。我们提出了IsoScore：一种量化点云均匀利用环境向量空间的程度的新工具。使用严格设计的测试，我们证明了IsoScore是文献中唯一可用的工具，可以准确地测量向量空间中跨维度均匀分布的方差。此外，我们使用IsoScore来挑战NLP文献中最近的一些结论，这些结论是使用各向同性的脆性指标得出的。我们提醒未来的研究不要使用现有的工具来测量背景嵌入空间中的各向同性，因为由此得出的结论将具有误导性或完全不准确。
The recent success of distributed word representations has led to an increased interest in analyzing the properties of their spatial distribution. Several studies have suggested that contextualized word embedding models do not isotropically project tokens into vector space. However, current methods designed to measure isotropy, such as average random cosine similarity and the partition score, have not been thoroughly analyzed and are not appropriate for measuring isotropy. We propose IsoScore: a novel tool that quantifies the degree to which a point cloud uniformly utilizes the ambient vector space. Using rigorously designed tests, we demonstrate that IsoScore is the only tool available in the literature that accurately measures how uniformly distributed variance is across dimensions in vector space. Additionally, we use IsoScore to challenge a number of recent conclusions in the NLP literature that have been derived using brittle metrics of isotropy. We caution future studies from using existing tools to measure isotropy in contextualized embedding space as resulting conclusions will be misleading or altogether inaccurate.