Deep neural networks

点击翻译按钮获取中文摘要

The success of deep neural networks (DNNs) is undeniable: the recent boom of large language models (LLMs), such as OpenAI's GPT models, Google's PaLM and Gemini models, and Anthropic's Bard, has alone been highlighted as a key area of interest and potential by the House of Lords and the Alan Turing Institute. However, the resource requirements for deep learning models is significant, almost prohibitively so in the case of large transformer models. This renders many applications infeasible: for example, many low-memory edge devices can not run inference for even a moderately-sized LLM.Effort has been made to resolve the computational issues of DNNs, increasing the efficiency of the model while still maintaining accuracy, largely through compression methods. One such method is pruning, removing parameters from the full network, which allows a network to be reduced in size by upwards of 90% while still scoring high (relative) accuracy. Another compression method is matrix sketching, low-dimension or -rank approximations of the internal matrices of parameters, which can yield similar results in some settings.While showing promise, these and other compression methods are yet to make DNNs sufficiently computationally efficient for many applications and each method has its own issues and limitations. Furthermore, most methods and associated theory are tailored for particular models, e.g. convolutional neural networks, and so newer models, such as transformers, lack robust and mathematically guaranteed methods. As such, this project will examine and extend compression methods for DNNs, focussing on matrix sketching methods for transformer models.Transformers, with their many long context-length matrices, are prime targets for matrix sketching methods and recent developments show that matrix sketching can reduce computational cost of transformer models by 50%. However, this reduction is low in comparison to the state-of-the-art for other models and furthermore the methods are tailored for inference and so do not apply to training. We will seek to develop new matrix sketching methods that can achieve greater reductions for transformer models, achieving this by leveraging and progressing new results regarding over-parameterisation and generalisation theory of DNNs and hashing-based matrix sketching methods. We will then apply the underlying principles of these methods to develop analogous variants for the training process of transformer models. With these methods developed and their performance proven, we can assess whether they can be generalised further to other models, potentially placing them in an overarching framework of matrix sketching methods for DNNs, and the implications of such parameter reductions on the over-parameterisation regime of DNNs.This project is supervised by Professor Jared Tanner and Professor Coralia Cartis. Additionally, we may collaborate with Dr. Shiwei Liu, particularly on what the current empirical results demonstrate and suggest. Finally, the project's direction on reducing computational requirements of DNNs is proposed by Advanced Micro Devices Inc.Completion of this project will entail improved methods and understanding of the compression of DNNs. This will reduce their computational cost, thereby making them more accessible and applicable. As such, this project falls within the EPSRC Mathematical Sciences, the Numerical Analysis, and the Non-Linear Systems areas of research.