Deep neural networks

深度神经网络（DNN）的成功是不可否认的：最近大型语言模型（LLM）的繁荣，如OpenAI的GPT模型，谷歌的PaLM和Gemini模型，以及Anthropic的Bard，已经被上议院和艾伦图灵研究所强调为感兴趣和潜力的关键领域。然而，深度学习模型的资源需求是显著的，在大型Transformer模型的情况下几乎是禁止的。这使得许多应用程序不可行：例如，许多低内存边缘设备甚至无法为中等大小的LLM运行推理。人们已经努力解决DNN的计算问题，提高模型的效率，同时仍然保持准确性，主要是通过压缩方法。一种这样的方法是修剪，从整个网络中删除参数，这允许网络的大小减少90%以上，同时仍然具有高（相对）准确性。另一种压缩方法是矩阵草图，参数内部矩阵的低维或秩近似，在某些情况下可以产生类似的结果。虽然显示出希望，但这些和其他压缩方法尚未使DNN在许多应用中具有足够的计算效率，并且每种方法都有自己的问题和局限性。此外，大多数方法和相关理论都是为特定模型定制的，例如卷积神经网络，因此较新的模型，如变压器，缺乏鲁棒性和数学保证的方法。因此，本项目将研究和扩展DNN的压缩方法，重点是Transformer模型的矩阵草图方法。变压器具有许多长上下文长度矩阵，是矩阵草图方法的主要目标，最近的发展表明，矩阵草图可以将Transformer模型的计算成本降低50%。然而，与其他模型的最新技术相比，这种减少是低的，此外，这些方法是为推理量身定制的，因此不适用于训练。我们将寻求开发新的矩阵草图绘制方法，可以实现更大的减少Transformer模型，实现这一目标，通过利用和进步的新成果，过度参数化和泛化理论的DNN和基于散列的矩阵草图绘制方法。然后，我们将应用这些方法的基本原理，为Transformer模型的训练过程开发类似的变体。随着这些方法的开发和性能的证明，我们可以评估它们是否可以进一步推广到其他模型，可能将它们置于DNN矩阵草图方法的总体框架中，以及这些参数减少对DNN过度参数化制度的影响。此外，我们可以与刘世伟博士合作，特别是在目前的实证结果表明和建议。最后，Advanced Micro Devices Inc.提出了减少DNN计算需求的项目方向。该项目的完成将需要改进DNN压缩方法和理解。这将降低它们的计算成本，从而使它们更容易获得和适用。因此，该项目属于EPSRC数学科学，数值分析和非线性系统研究领域的福尔斯。

The success of deep neural networks (DNNs) is undeniable: the recent boom of large language models (LLMs), such as OpenAI's GPT models, Google's PaLM and Gemini models, and Anthropic's Bard, has alone been highlighted as a key area of interest and potential by the House of Lords and the Alan Turing Institute. However, the resource requirements for deep learning models is significant, almost prohibitively so in the case of large transformer models. This renders many applications infeasible: for example, many low-memory edge devices can not run inference for even a moderately-sized LLM.Effort has been made to resolve the computational issues of DNNs, increasing the efficiency of the model while still maintaining accuracy, largely through compression methods. One such method is pruning, removing parameters from the full network, which allows a network to be reduced in size by upwards of 90% while still scoring high (relative) accuracy. Another compression method is matrix sketching, low-dimension or -rank approximations of the internal matrices of parameters, which can yield similar results in some settings.While showing promise, these and other compression methods are yet to make DNNs sufficiently computationally efficient for many applications and each method has its own issues and limitations. Furthermore, most methods and associated theory are tailored for particular models, e.g. convolutional neural networks, and so newer models, such as transformers, lack robust and mathematically guaranteed methods. As such, this project will examine and extend compression methods for DNNs, focussing on matrix sketching methods for transformer models.Transformers, with their many long context-length matrices, are prime targets for matrix sketching methods and recent developments show that matrix sketching can reduce computational cost of transformer models by 50%. However, this reduction is low in comparison to the state-of-the-art for other models and furthermore the methods are tailored for inference and so do not apply to training. We will seek to develop new matrix sketching methods that can achieve greater reductions for transformer models, achieving this by leveraging and progressing new results regarding over-parameterisation and generalisation theory of DNNs and hashing-based matrix sketching methods. We will then apply the underlying principles of these methods to develop analogous variants for the training process of transformer models. With these methods developed and their performance proven, we can assess whether they can be generalised further to other models, potentially placing them in an overarching framework of matrix sketching methods for DNNs, and the implications of such parameter reductions on the over-parameterisation regime of DNNs.This project is supervised by Professor Jared Tanner and Professor Coralia Cartis. Additionally, we may collaborate with Dr. Shiwei Liu, particularly on what the current empirical results demonstrate and suggest. Finally, the project's direction on reducing computational requirements of DNNs is proposed by Advanced Micro Devices Inc.Completion of this project will entail improved methods and understanding of the compression of DNNs. This will reduce their computational cost, thereby making them more accessible and applicable. As such, this project falls within the EPSRC Mathematical Sciences, the Numerical Analysis, and the Non-Linear Systems areas of research.