Deep neural networks

深度神经网络（dnn）的成功是不可否认的：最近兴起的大型语言模型（llm），如OpenAI的GPT模型，b谷歌的PaLM和Gemini模型，以及Anthropic的Bard模型，已经被上议院和艾伦图灵研究所（Alan Turing Institute）强调为一个感兴趣和有潜力的关键领域。然而，深度学习模型的资源需求是显著的，在大型变压器模型的情况下几乎是令人望而却步的。这使得许多应用程序不可行：例如，许多低内存边缘设备甚至不能为中等大小的LLM运行推理。人们已经努力解决深度神经网络的计算问题，在保持精度的同时提高模型的效率，主要是通过压缩方法。其中一种方法是修剪，从整个网络中删除参数，这使得网络的大小减少了90%以上，同时仍然获得较高的（相对）准确性。另一种压缩方法是矩阵素描，对参数的内部矩阵进行低维或秩近似，在某些情况下可以产生类似的结果。虽然这些和其他压缩方法显示出希望，但对于许多应用程序，这些方法尚未使dnn具有足够的计算效率，每种方法都有自己的问题和局限性。此外，大多数方法和相关理论都是为特定模型量身定制的，例如卷积神经网络，因此较新的模型，如变压器，缺乏鲁棒性和数学上保证的方法。因此，该项目将研究和扩展dnn的压缩方法，重点是变压器模型的矩阵草图绘制方法。变压器具有许多较长的上下文长度矩阵，是矩阵素描方法的主要目标，最近的发展表明，矩阵素描可以将变压器模型的计算成本降低50%。然而，与其他模型的最新技术相比，这种减少是低的，而且这些方法是为推理量身定制的，因此不适用于训练。我们将寻求开发新的矩阵草图方法，通过利用和推进关于dnn的过参数化和泛化理论以及基于哈希的矩阵草图方法的新结果，可以实现变压器模型的更大减少。然后，我们将应用这些方法的基本原理来开发变压器模型训练过程的类似变体。随着这些方法的发展和它们的性能证明，我们可以评估它们是否可以进一步推广到其他模型，可能将它们置于dnn矩阵草图方法的总体框架中，以及这种参数缩减对dnn过度参数化制度的影响。本项目由Jared Tanner教授和Coralia Cartis教授指导。此外，我们可能会与刘世伟博士合作，特别是在目前的实证结果证明和建议。最后，Advanced Micro Devices inc .提出了降低深度神经网络计算需求的项目方向。该项目的完成将需要改进方法和对深度神经网络压缩的理解。这将降低它们的计算成本，从而使它们更易于访问和适用。因此，该项目属于EPSRC数学科学、数值分析和非线性系统研究领域。

The success of deep neural networks (DNNs) is undeniable: the recent boom of large language models (LLMs), such as OpenAI's GPT models, Google's PaLM and Gemini models, and Anthropic's Bard, has alone been highlighted as a key area of interest and potential by the House of Lords and the Alan Turing Institute. However, the resource requirements for deep learning models is significant, almost prohibitively so in the case of large transformer models. This renders many applications infeasible: for example, many low-memory edge devices can not run inference for even a moderately-sized LLM.Effort has been made to resolve the computational issues of DNNs, increasing the efficiency of the model while still maintaining accuracy, largely through compression methods. One such method is pruning, removing parameters from the full network, which allows a network to be reduced in size by upwards of 90% while still scoring high (relative) accuracy. Another compression method is matrix sketching, low-dimension or -rank approximations of the internal matrices of parameters, which can yield similar results in some settings.While showing promise, these and other compression methods are yet to make DNNs sufficiently computationally efficient for many applications and each method has its own issues and limitations. Furthermore, most methods and associated theory are tailored for particular models, e.g. convolutional neural networks, and so newer models, such as transformers, lack robust and mathematically guaranteed methods. As such, this project will examine and extend compression methods for DNNs, focussing on matrix sketching methods for transformer models.Transformers, with their many long context-length matrices, are prime targets for matrix sketching methods and recent developments show that matrix sketching can reduce computational cost of transformer models by 50%. However, this reduction is low in comparison to the state-of-the-art for other models and furthermore the methods are tailored for inference and so do not apply to training. We will seek to develop new matrix sketching methods that can achieve greater reductions for transformer models, achieving this by leveraging and progressing new results regarding over-parameterisation and generalisation theory of DNNs and hashing-based matrix sketching methods. We will then apply the underlying principles of these methods to develop analogous variants for the training process of transformer models. With these methods developed and their performance proven, we can assess whether they can be generalised further to other models, potentially placing them in an overarching framework of matrix sketching methods for DNNs, and the implications of such parameter reductions on the over-parameterisation regime of DNNs.This project is supervised by Professor Jared Tanner and Professor Coralia Cartis. Additionally, we may collaborate with Dr. Shiwei Liu, particularly on what the current empirical results demonstrate and suggest. Finally, the project's direction on reducing computational requirements of DNNs is proposed by Advanced Micro Devices Inc.Completion of this project will entail improved methods and understanding of the compression of DNNs. This will reduce their computational cost, thereby making them more accessible and applicable. As such, this project falls within the EPSRC Mathematical Sciences, the Numerical Analysis, and the Non-Linear Systems areas of research.