A Lebesgue Integral based Approximation for Language Modelling

近年来，基于深度学习（DL）的自然语言处理（NLP）技术引起了人们的极大兴趣。当前的SOTA语言模型，也就是基于变换器的语言模型通常假定给定单词的表示可以通过在凸船体中对其相关上下文的内插来捕获。然而，最近已经表明，在高维空间中，插值几乎肯定不会发生，无论数据流形的内在维度。由这种基于转换器的语言模型生成的表示将收敛到一个密集的锥状超空间中，该超空间通常是不连续的，具有许多不相邻的簇。为了克服大多数基于DL的NLP模型中当前方法的局限性，该项目旨在部署Lebesgue积分，它可以被定义为分区之间的积分集合（即，不连续特征聚类），通过自动识别这种不连续集合的边界来近似有限可测量集合中给定输入词特征的聚类的后验分布，这反过来可以帮助生成更好的解释并量化不确定性。通过我们提出的基于Lebesgue积分的近似，输入文本将由两个属性来表征：一个指示符向量，其编码其在聚类中的成员资格（即，可测量集合），以及另一个连续特征表示，用于更好地捕获其语义含义以用于下游任务。这不仅可以更忠实地近似NLP中输入文本分布中常见的可数不连续性，而且还可以学习人类更好理解的文本表示。

Deep learning (DL) based Natural Language Processing (NLP) technologies have attracted significant interest in recent years. The current SOTA language models, a.k.a. transformer-based language models, typically assume that the representation of a given word can be captured by the interpolation of its related context in a convex hull. However, it has recently been shown that in high-dimensional spaces, the interpolation almost surely never occurs regardless of the underlying intrinsic dimension of the data manifold. The representations generated by such transformer-based language models will converge into a dense cone-like hyperspace which is often discontinuous with many nonadjacent clusters. To overcome the limitation of current methods in most DL-based NLP models, this project aims to deploy Lebesgue integral, which can be defined as an ensemble of integrals among partitions (i.e., discontinuous feature clusters), to approximate the posterior distributions of clusters given input word features in finite measurable sets by automatically identifying the boundary of such discontinuous set, which in turn could help to generate better interpretations and quantify the uncertainty. By our proposed Lebesgue integral based approximation, the input text will be characterised by two properties: an indicator vector encoding its membership in clusters (i.e., measurable sets), and another continuous feature representation for better capturing its semantic meaning for downstream tasks. This not only allows for a more faithful approximation of commonly observed countably discontinuities in distributions of input text in NLP, but also enables learning text representations that are better understood by humans.