Cross-Lingual Knowledge Representation and Alignment in LLMs

大型语言模型（LLM）在许多下游任务中表现出出色的性能。然而，它们表现出依赖于语言的不同能力，在高资源语言上具有优化的性能，限制了低资源语言任务的有效性。这种局限性主要归因于语言之间固有的知识基础不平衡，这种不平衡主要表现在两个方面：知识差异和跨语言知识的重复性。知识差距指的是，当以不同的语言提出相同的问题时，法学硕士可能会提供不同的回答。另一方面，跨语言知识的重复性涉及到模型在一种特定语言上更新其知识时不同步其他语言的这种增量的情况。这些挑战不容易通过应用神经机器翻译（NMT）技术来解决，因为NMT模型（LLM或非LLM）继承了相同的语言不平衡问题。为了确保不同语言输入的一致模型性能，我们的目标是探索LLM中的跨语言知识表示和对齐。该研究涉及理解跨语言知识表示的特征，并将这些见解应用于开发跨语言知识对齐的方法。我们的研究方法分为两个主要途径：外部转型和内部调整。外部转换涉及在LLM外部集成多语言转换器，以便于将低资源语言表示转换为高资源语言表示，即，英语内部对齐（Internal alignment）是指通过对跨语言知识的特征进行推断，并对神经网络进行特定的修改，以保证跨语言知识表示的一致性。我们定义了三个基本的研究问题：RQ1：LLM中跨语言知识表示的本质是什么？研究方向包括开发跨语言探针，以揭示LLM的内部工作机制。我们希望所获得的见解指导我们开发工具来解决语言之间的知识基础不平衡。RQ2：什么是调整内部跨语言知识表示的最佳方法？重点是开发知识调整技术，以有效地将知识从高资源语言转移到低资源语言。研究可能包括引入神经网络来跨各种语言传递知识表示。RQ3：我们如何通过引入外部结构来调整跨语言知识？我们可以引入外部多语言知识表示转换器来将表示从其他语言转换为高资源语言（即，英语），确保LLM跨语言设置的一致性。

Large language models (LLMs) have demonstrated outstanding performance across many downstream tasks. However, they manifest language-dependent disparate capabilities with optimised performances on high-resource language, limiting the effectiveness on low-resource language tasks. This limitation is primarily attributed to the inherent knowledge grounding imbalance between languages, which manifests in two key aspects: knowledge disparities and cross-lingual knowledge asynchronicity. Knowledge disparities refer to the fact that LLMs may provide different responses when presented with the same question posed in different languages. Cross-lingual knowledge asynchronicity, on the other hand, relates to situations where a model updating its knowledge on one particular language does not synchronize such increments for other languages. These challenges are not readily addressed through applying Neural Machine Translation (NMT) techniques, as an NMT model (LLMs or not) inherits the same issues of language imbalance.To ensure consistent model performance for inputs in diverse languages, we aim to explore cross-lingual knowledge representation and alignment in LLMs. The research involves understanding the characteristics of cross-lingual knowledge representation and applying the insights to develop methods to align cross-lingual knowledge. Our research methods fall into two main avenues: external transformation and internal alignment. External transformation involves integrating a multilingual converter outside LLMs to facilitate the conversion of low-resource language representations into high-resource language representations, i.e., English. Internal alignment is to infer the feature of cross-lingual knowledge and apply specific neural network modifications to ensure the consistency of cross-knowledge representations.We define three fundamental research questions:RQ1: What is the nature of cross-lingual knowledge representation within LLMs? The line of inquiry involves developing cross-lingual probes to unveil the inner working mechanism of LLMs. We hope the gained insights guide us to develop instruments to solve knowledge grounding imbalance between languages.RQ2: What is the optimal way to align internal cross-lingual knowledge representation? The focus is on developing knowledge alignment techniques to effectively transfer knowledge from a high-resource language to its low-resource counterparts. The research may include introducing neural networks to transfer knowledge representations across various languages.RQ3: How can we align cross-lingual knowledge by bringing in external structures? We may introduce an external multilingual knowledge representation converter to transform representations from other languages into a high-resource language (i.e., English), ensuring LLMs consistency across language settings.