Cross-Lingual Knowledge Representation and Alignment in LLMs

点击翻译按钮获取中文摘要

Large language models (LLMs) have demonstrated outstanding performance across many downstream tasks. However, they manifest language-dependent disparate capabilities with optimised performances on high-resource language, limiting the effectiveness on low-resource language tasks. This limitation is primarily attributed to the inherent knowledge grounding imbalance between languages, which manifests in two key aspects: knowledge disparities and cross-lingual knowledge asynchronicity. Knowledge disparities refer to the fact that LLMs may provide different responses when presented with the same question posed in different languages. Cross-lingual knowledge asynchronicity, on the other hand, relates to situations where a model updating its knowledge on one particular language does not synchronize such increments for other languages. These challenges are not readily addressed through applying Neural Machine Translation (NMT) techniques, as an NMT model (LLMs or not) inherits the same issues of language imbalance.To ensure consistent model performance for inputs in diverse languages, we aim to explore cross-lingual knowledge representation and alignment in LLMs. The research involves understanding the characteristics of cross-lingual knowledge representation and applying the insights to develop methods to align cross-lingual knowledge. Our research methods fall into two main avenues: external transformation and internal alignment. External transformation involves integrating a multilingual converter outside LLMs to facilitate the conversion of low-resource language representations into high-resource language representations, i.e., English. Internal alignment is to infer the feature of cross-lingual knowledge and apply specific neural network modifications to ensure the consistency of cross-knowledge representations.We define three fundamental research questions:RQ1: What is the nature of cross-lingual knowledge representation within LLMs? The line of inquiry involves developing cross-lingual probes to unveil the inner working mechanism of LLMs. We hope the gained insights guide us to develop instruments to solve knowledge grounding imbalance between languages.RQ2: What is the optimal way to align internal cross-lingual knowledge representation? The focus is on developing knowledge alignment techniques to effectively transfer knowledge from a high-resource language to its low-resource counterparts. The research may include introducing neural networks to transfer knowledge representations across various languages.RQ3: How can we align cross-lingual knowledge by bringing in external structures? We may introduce an external multilingual knowledge representation converter to transform representations from other languages into a high-resource language (i.e., English), ensuring LLMs consistency across language settings.