The Role of Neural Models in (Constrained) Natural Language Generation Built on the mathematical foundations laid out by Markov [1], n-gram language

n-gram语言模型建立在Markov[1]提出的数学基础上，赋予一个单词一个概率分布，这个概率分布取决于它的上下文（它前面的n个单词）。虽然这些模型导致了语言技术的进步，如语音识别b[2]和机器翻译[3,4]，但它们的参数化的基数随着上下文长度呈指数增长。这限制了它们的适用性。递归神经网络可以模拟任意长度的上下文b[5]，这使得这些模型在许多语言任务b[6]中被广泛采用。克服了循环架构的计算效率限制，b[7]中提出的神经“变压器”架构导致了一系列神经语言模型的发展，这些模型在各种自然语言任务中达到了最先进的性能[8,9,10]。该项目旨在通过研究如何使用神经语言模型将用户目标整合到自然语言生成中，来推进基于人工智能（AI）的自然语言技术。这些目标可以通过与系统的交互来表达，就像对话式AI一样。受神经模型被用来取代复杂模型管道的最新发展的启发[11,12,13]，该项目将探索并提供新颖的方法，使这些系统能够：更好地估计、跟踪和根据用户意图调整或自适应其输出，以满足新的用户目标。目标也可以通过语言工程师希望嵌入生成任务的约束来表达。例如，对于最先进的自动翻译系统来说，在语言中产生性别屈折变化（后者取决于人类指涉者的社会性别）是一项挑战，最近的研究表明，在神经模型中嵌入这种约束是非常重要的。解决这个问题不仅仅是为了减轻性别偏见，还可以采用类似的方法来限制神经系统生成性别中立的翻译。该项目旨在采取广泛的方法来推进用户约束语言生成的最新技术，调查适当的数据源及其最佳表示架构变化，以适应新的/更丰富的输入数据表示新的训练方法，包括与其他系统结合的新目标和训练，以及考虑约束适应技术的新解码过程。参考文献[1]Markov， a . a .（1913）。Essai d'une recherche statisque sur le texte du roman“Eugene Onegin”插图la liaison des epreuve en chain（“Eugene Onegin”文本的统计调查示例，说明样本之间的依赖性）。圣-帕姆斯堡科学与农业学院学报，7,153-162.[j]Povey， D, & Woodland， P. C.（2002年5月）。最小电话误差和i -平滑改进判别训练。2002年IEEE声学、语音与信号处理国际学术会议（第1卷，第1 -105页）。IEEE。[3]蒋D.（2005, 6月）。基于分层短语的统计机器翻译模型。计算语言学协会第43届年会论文集（ACL'05） （pp. 263-270）。bbb10 de Gispert， A., Iglesias， G., Blackwood， G.， R. Banga， E., & Byrne， W.（2010）。基于加权有限状态换能器和浅n语法的分层短语翻译。计算语言学，36(3),505-533Mikolov， T., Karafiát， M., Burget， L., cernoky， J., & Khudanpur， S.（2010）。基于递归神经网络的语言模型。国际语音交流协会第十一届年会（第1045-1048页）Jurafsky， D., & Martin， J. H.,（未注明日期）。用神经网络处理序列。语音和语言处理导论

Built on the mathematical foundations laid out by Markov [1], n-gram language models endow a word with a probability distribution that depends on its context (the n words preceding it). While these models led to advances in language technologies as diverse as speech recognition [2] and machine translation [3, 4], the cardinality of their parametrisation grows exponentially with the context length. This limits their applicability. Recurrent neural networks can model arbitrary length contexts [5], which led to the widespread adoption of these models in many language tasks [6]. Overcoming the computational efficiency limitations of recurrent architectures, the neural "transformer" architecture proposed in [7] led to development of a range of neural language models that achieve state-of-the-art performance in a variety of natural language tasks [8, 9, 10].This project aims to advance artificial intelligence (AI) based technologies for natural language by investigating how neural language models can be employed to incorporate user goals in natural language generation. Such goals might be expressed through interaction with the system, as is the case in conversational AI. Inspired by recent developments in the field where neural models have been employed to replace complex model pipelines [11, 12, 13], the project will explore and provide novel methods that will allow these systems to: better estimate, track and ground their output in user intent be adapted or self-adapt to satisfy new user goalsGoals might also be expressed through constraints that a language engineer would like to embed in the generation task. For example, generation of gender inflections in languages where the latter depends on the social gender of a human referent is challenging for state-of-the-art automatic translation systems [14] and recent work has shown that embedding such constraints in neural models is non-trivial [15]. Solving this problem extends beyond gender bias mitigation, as similar approaches might be pursued to constrain neural systems to generate gender-neutral translations. This project aims to take a broad approach to advancing state-of-the-art in user-constrained language generation, investigating: appropriate data sources and their optimal representation architectural changes necessary to accommodate new/richer input data representations new training methodologies, including novel objectives and training in conjunction with other systems novel decoding processes that account for constraints adaptation techniquesReferences[1] Markov, A. A. (1913). Essai d'une recherche statistique sur le texte du roman "Eugene Onegin" illustrant la liaison des epreuve en chain ('Example of a statistical investigation of the text of "Eugene Onegin" illustrating the dependence between samples in chain'). Izvistia Imperatorskoi Akademii Nauk (Bulletin de l'Academie Impériale des Sciences de St.-Pétersbourg), 7, 153-162.[2] Povey, D., & Woodland, P. C. (2002, May). Minimum phone error and I-smoothing for improved discriminative training. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. 1, pp. I-105). IEEE.[3] Chiang, D. (2005, June). A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL'05) (pp. 263-270).[4] de Gispert, A., Iglesias, G., Blackwood, G., R. Banga, E., & Byrne, W. (2010). Hierarchical phrase-based translation with weighted finite-state transducers and shallow-n grammars. Computational linguistics, 36(3), 505-533.[5] Mikolov, T., Karafiát, M., Burget, L., Cernocky, J., & Khudanpur, S. (2010). Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association (pp. 1045-1048).[6] Jurafsky, D., &, Martin, J. H., (n.d). Sequence Processing with Neural Networks. In Speech and Language Processing: An Introd