A Lebesgue Integral based Approximation for Language Modelling

近年来，基于深度学习（DL）的自然语言处理（NLP）技术引起了人们的极大兴趣。当前的SOTA语言模型，也称为基于转换器的语言模型，通常假设给定单词的表示可以通过在凸包中插入相关上下文来捕获。然而，最近的研究表明，在高维空间中，无论数据流形的内在维数是多少，插值几乎肯定不会发生。由这种基于变换的语言模型生成的表示将收敛成密集的锥状超空间，该空间通常是不连续的，有许多不相邻的簇。为了克服大多数基于dl的NLP模型中现有方法的局限性，本项目旨在部署Lebesgue积分，该积分可以定义为分区（即不连续特征聚类）之间的积分集合，通过自动识别这种不连续集的边界来近似有限可测量集中给定输入词特征的聚类的后测分布，从而有助于生成更好的解释和量化不确定性。通过我们提出的基于勒贝格积分的近似，输入文本将由两个属性来表征：一个指示向量编码其在聚类（即可测量集）中的隶属关系，另一个连续特征表示用于更好地捕获其下游任务的语义。这不仅可以更忠实地近似NLP中输入文本分布中通常观察到的可数不连续，而且还可以学习人类更好理解的文本表示。

Deep learning (DL) based Natural Language Processing (NLP) technologies have attracted significant interest in recent years. The current SOTA language models, a.k.a. transformer-based language models, typically assume that the representation of a given word can be captured by the interpolation of its related context in a convex hull. However, it has recently been shown that in high-dimensional spaces, the interpolation almost surely never occurs regardless of the underlying intrinsic dimension of the data manifold. The representations generated by such transformer-based language models will converge into a dense cone-like hyperspace which is often discontinuous with many nonadjacent clusters. To overcome the limitation of current methods in most DL-based NLP models, this project aims to deploy Lebesgue integral, which can be defined as an ensemble of integrals among partitions (i.e., discontinuous feature clusters), to approximate the posterior distributions of clusters given input word features in finite measurable sets by automatically identifying the boundary of such discontinuous set, which in turn could help to generate better interpretations and quantify the uncertainty. By our proposed Lebesgue integral based approximation, the input text will be characterised by two properties: an indicator vector encoding its membership in clusters (i.e., measurable sets), and another continuous feature representation for better capturing its semantic meaning for downstream tasks. This not only allows for a more faithful approximation of commonly observed countably discontinuities in distributions of input text in NLP, but also enables learning text representations that are better understood by humans.