Efficient deep learning in resource constrained environments

该项目属于EPSRC信息和通信（ICT）研究领域。“资源受限环境下的高效深度学习”项目旨在进一步加快和减少机器学习（ML）模型，特别是深度神经网络的内存占用。即使在昂贵的硬件上运行，神经网络的速度也是学术界和工业界普遍提出的一个问题。ML从业者希望将强大的模型部署到低功耗的硬件上，比如移动设备，而ML研究人员希望在模型开发过程中快速迭代，使其更具交互性和生产力。ML系统通常具有以下组件：(a)定义目标设备如何运行模型的硬件/软件实现，(b)定义如何处理输入数据的模型（通常取决于数据的结构），以及(c)算法（损失函数优化器），它指示如何在输入数据上训练模型。这些组件在很大程度上被看作是独立的，使模型开发人员能够混合和匹配不同的选项来解决他们的问题。因此，现有的模型性能优化研究往往是孤立地关注各个部分。该项目的目的是开发方法，通过绕过所述组件之间的边界，并以联合和相互通知的方式在整个机器学习系统中应用优化，从而获得进一步的速度和内存收益。例如，当前用于深度神经网络的软件工具包和硬件加速器通常很少或根本没有对正在运行的神经网络的整体架构或正在处理的数据结构进行假设。类似地，模型架构师倾向于对底层硬件做很少或不做任何假设，从而错过了潜在的速度或内存增益。具体而言，该项目的目标是开发神经网络的实施、设计和训练方法，以充分利用目标设备的能力和约束以及手头问题的数据结构。这对于在时间压力下处理大量数据或在有限硬件环境中操作的工业应用和科学领域非常有用，例如基因组学，高能物理，金融数据分析等。更雄心勃勃的研究成果之一也可能是机器学习算法，当提供数据和硬件约束的样本时自动发现这种优化。该项目与人工智能EPSRC ICT研究方向一致；没有计划进行工业合作。

This project falls within the EPSRC Information and Communication (ICT) research area.The project "Efficient deep learning in resource constrained environments" aims to make further advances in accelerating and reducing memory footprint of machine learning (ML) models and, in particular, deep neural networks. The slowness of neural networks, even when run on expensive hardware, is commonly raised as an issue in both academia and industry. ML practitioners would like to deploy powerful models to low-powered hardware, such as mobile devices, and ML researchers would like to iterate quickly during model development, making it more interactive and productive.An ML system typically has the following components:(a) the hardware/software implementation which defines how the target device runs the model,(b) the model which defines how input data is processed (often depends on the structure of the data), and(c) the algorithm (loss function optimizer) which dictates how the model is trained on the input data.These components are largely seen as independent, enabling model developers to mix and match different options for their problem. As a result, existing research in model performance optimization tends to focus on each part in isolation.The aim of this project is to develop methods that extract further speed and memory gains by bypassing boundaries between said components, and applying optimisations throughout the entire ML system in a joint and mutually-informed way. For example, current software toolkits and hardware accelerators for deep neural networks often make little or no assumptions about the overall architecture of the neural network being ran or the structure of the data it is processing. Similarly, model architects tend to make little or no assumptions about the underlying hardware, missing out on potential speed or memory gains.Concretely, the objective of the project is to develop neural network implementation, design and training methodology that make the best use of capabilities and constraints of the target device as well as the structure of the data for the problem at hand. This would be of great use for industrial applications and fields of science which have to deal with vast quantities of data under time pressure or operate in environments restricted hardware, such as genomics, high energy physics, financial data analysis, etc.One of more ambitious research outcomes may also be a machine learning algorithm to discover such optimisations automatically when provided with a sample of the data and hardware constraints.The project aligns with the Artificial Intelligence EPSRC ICT research direction; no industrial collaborations are planned.