MultiTasking and Continual Learning for Audio Sensing Tasks on Resource-Constrained Platforms

深度学习（机器学习的一种形式）在解决计算机视觉、自然语言处理和语音处理领域的许多复杂任务方面非常成功，因为它能够从原始数据中学习有效的表示。这一成功推动了在资源受限的计算平台（如微控制器（mcu））上部署深度学习模型的需求，因为它们高度精确，并且在本地设备上部署它们将增强用户数据的隐私性。廉价传感器和物联网（IoT）设备的激增将进一步推动这一需求，最近不断增长的TinyML（将机器/深度学习放在微型设备上）领域的趋势就是一个迹象。最近，在实现在资源受限设备上有效部署深度学习模型的目标方面取得了重大进展。然而，重点仍然局限于有效地解决单个任务。此外，模型是静态的；他们不能随着时间的推移而学习。我们认为现在是时候超越静态的“一旦学会并部署”深度学习模型了。需要具备多任务处理和持续学习的能力，以适应未见过的变化、学习新信息和处理多个不同的应用程序。然而，在资源受限的设备上容纳这些功能是极具挑战性的：有限的内存和计算能力。为此，该项目旨在开发一系列技术，使深度学习模型能够动态学习，并在资源受限的设备上有效地解决多个任务（低延迟、低功耗）。总的来说，项目目标是：(a)设计一种优化的内存管理方案，使多个深度学习模型保持在设备的可用内存中；(b)设计一种新颖的调度策略，可以有效地为应用程序分配设备上可用处理内核的工作量，并并行执行任务；最后(c)提出一种将持续学习与少量学习范式相结合的方法，使模型能够在设备上使用少量注释数据进行连续学习。开发的技术将在嵌入式音频平台上进行测试，用于各种音频传感任务，如关键字识别、音频场景分析（定位、场景分类、声音分类）和语音增强。对音频的关注是由于其在许多核心应用中的地位日益突出，例如Alexa等家庭中心、生态监测、疾病诊断、预防性维护、可听设备和助听器等辅助设备。这项工作产生的创新将带来许多好处。首先，开发可在设备上运行的高效深度学习解决方案将节省电力，从而降低数字碳足迹，建立一个更加可持续的社会，并为英国实现2050年零网络的使命做出贡献。本地执行进一步增强了用户隐私，因为数据永远不会离开设备。在没有网络通信或网络通信费用昂贵的地方，如LMIC（低收入和中等收入国家），这也是一个关键的好处。其次，创建这样的解决方案意味着用户可以在日常生活中享受深度学习为许多有用和无处不在的计算应用程序提供的优势（通常是高精度、高性能）。最后，通过在设备上不断学习，我们将离能够反映真正人类智能的机器更近一步。

Deep learning (a form of machine learning) has been highly successful in solving many complex tasks in the domain of computer vision, natural language processing and speech processing due to its ability to learn effective representations from raw data. This success is pushing demands to deploy deep learning models on more resource-constrained computing platforms such as Micro Controller Units (MCUs) as they are highly accurate and deploying them locally on the device will give enhanced privacy to user data. Proliferation of cheap sensors and IoT (Internet of Things) devices will further fuel this demand and the recent trends in ever growing field of TinyML (putting machine/deep learning on tiny devices) is an indication thereof. Recently, significant gains have been made in realizing the goal of deploying deep learning models efficiently on resource-constrained devices. However, the focus is still limited to solving single tasks efficiently. Additionally, the models are static; they cannot learn with time. We think it is time to go beyond the static "once learnt and deploy" deep learning models. The ability to multi-task and learn continuously is required to adapt to unseen changes, learn new information, and handle multiple disparate applications. However, accommodating these abilities on resource-constrained devices is extremely challenging: limited memory and compute power. To this end, this project aims to develop a range of techniques that make deep learning models learn on the fly and solve multiple tasks efficiently (low latency, low power) on resource-constrained devices. Overall, the project goals are to: (a) design an optimal memory management scheme to keep multiple deep learning models in the available memory of the device, (b) devise novel scheduling strategies that can distribute the workload on available processing cores on device efficiently for an application and execute tasks in parallel, and finally (c) come up with a method that will combine continual learning with few shot learning paradigm to allow models to learn continuously with few annotated data on device. The developed techniques will be tested on an embedded audio platform for a variety of audio sensing tasks such as keyword spotting, audio scene analysis (localization, scene classification, sound classification), and speech enhancement. The focus on audio is due to its rising prominence in many core applications, such as home hubs like Alexa, ecological monitoring, disease diagnostics, preventive maintenance, hearables and accessibility devices such as hearing aids. The resulting innovations from this work will have numerous benefits. First of all, developing efficient deep learning solutions that can run on-device would save power leading to a lower digital carbon footprint, a far more sustainable society and contributing to the UK's mission of NetZero 2050. Local execution further leads to enhanced user privacy as data never leaves the device. This is also a key benefit at places where network communication is absent or can be expensive such as LMIC (Low- and middle-income countries). Secondly, creating such solutions means users can enjoy the advantages provided by deep learning (often high accuracy, high performance) for many useful and ubiquitous computing applications in their day to day lives. Finally, by doing continual learning on the device we will move one inch closer to machines that can reflect true human intelligence.