MultiTasking and Continual Learning for Audio Sensing Tasks on Resource-Constrained Platforms

深度学习是机器学习的一种形式，由于它能够从原始数据中学习有效的表示，在计算机视觉、自然语言处理和语音处理领域中已经成功地解决了许多复杂的任务。这种成功推动了在资源更有限的计算平台(如微控制器单元(MCU))上部署深度学习模型的需求，因为它们非常准确，并且将它们本地部署在设备上将增强用户数据的隐私。廉价传感器和物联网设备的激增将进一步推动这一需求，TinyML(将机器/深度学习放在微型设备上)领域的最新趋势就是一个迹象。最近，在实现在资源受限的设备上高效部署深度学习模型的目标方面取得了显著进展。然而，重点仍然局限于有效地解决单一任务。此外，这些模型是静态的；它们不能随时间学习。我们认为，现在是时候超越静态的“曾经学会并部署”的深度学习模式了。多任务处理和持续学习的能力是适应未知变化、学习新信息和处理多个不同应用程序所必需的。然而，在资源受限的设备上容纳这些功能是极具挑战性的：有限的内存和计算能力。为此，该项目旨在开发一系列技术，使深度学习模型在运行中学习，并在资源受限的设备上高效地解决多个任务(低延迟、低功耗)。总体来说，该项目的目标是：(A)设计一个最优的内存管理方案，将多个深度学习模型保存在设备的可用内存中；(B)设计新颖的调度策略，为应用程序有效地分配设备上可用处理核上的工作负载，并并行执行任务；(C)提出一种将持续学习和少镜头学习范例相结合的方法，使模型能够在设备上使用较少的注释数据进行连续学习。开发的技术将在嵌入式音频平台上进行测试，以执行各种音频感知任务，如关键词识别、音频场景分析(定位、场景分类、声音分类)和语音增强。对音频的关注是因为它在许多核心应用中日益突出，例如像Alexa这样的家庭中枢、生态监测、疾病诊断、预防性维护、听觉设备和助听器等无障碍设备。这项工作带来的创新将带来许多好处。首先，开发可以在设备上运行的高效深度学习解决方案将节省电力，从而降低数字碳足迹，实现更可持续的社会，并为英国的NetZero 2050使命做出贡献。本地执行进一步增强了用户隐私，因为数据永远不会离开设备。在LMIC(低收入和中等收入国家)等缺乏网络通信或可能很昂贵的地方，这也是一个关键的好处。其次，创建这样的解决方案意味着用户可以在日常生活中享受深度学习(通常是高精度、高性能)为许多有用和无处不在的计算应用程序提供的优势。最后，通过在设备上进行持续学习，我们将离能够反映真正人类智能的机器更近一英寸。

Deep learning (a form of machine learning) has been highly successful in solving many complex tasks in the domain of computer vision, natural language processing and speech processing due to its ability to learn effective representations from raw data. This success is pushing demands to deploy deep learning models on more resource-constrained computing platforms such as Micro Controller Units (MCUs) as they are highly accurate and deploying them locally on the device will give enhanced privacy to user data. Proliferation of cheap sensors and IoT (Internet of Things) devices will further fuel this demand and the recent trends in ever growing field of TinyML (putting machine/deep learning on tiny devices) is an indication thereof. Recently, significant gains have been made in realizing the goal of deploying deep learning models efficiently on resource-constrained devices. However, the focus is still limited to solving single tasks efficiently. Additionally, the models are static; they cannot learn with time. We think it is time to go beyond the static "once learnt and deploy" deep learning models. The ability to multi-task and learn continuously is required to adapt to unseen changes, learn new information, and handle multiple disparate applications. However, accommodating these abilities on resource-constrained devices is extremely challenging: limited memory and compute power. To this end, this project aims to develop a range of techniques that make deep learning models learn on the fly and solve multiple tasks efficiently (low latency, low power) on resource-constrained devices. Overall, the project goals are to: (a) design an optimal memory management scheme to keep multiple deep learning models in the available memory of the device, (b) devise novel scheduling strategies that can distribute the workload on available processing cores on device efficiently for an application and execute tasks in parallel, and finally (c) come up with a method that will combine continual learning with few shot learning paradigm to allow models to learn continuously with few annotated data on device. The developed techniques will be tested on an embedded audio platform for a variety of audio sensing tasks such as keyword spotting, audio scene analysis (localization, scene classification, sound classification), and speech enhancement. The focus on audio is due to its rising prominence in many core applications, such as home hubs like Alexa, ecological monitoring, disease diagnostics, preventive maintenance, hearables and accessibility devices such as hearing aids. The resulting innovations from this work will have numerous benefits. First of all, developing efficient deep learning solutions that can run on-device would save power leading to a lower digital carbon footprint, a far more sustainable society and contributing to the UK's mission of NetZero 2050. Local execution further leads to enhanced user privacy as data never leaves the device. This is also a key benefit at places where network communication is absent or can be expensive such as LMIC (Low- and middle-income countries). Secondly, creating such solutions means users can enjoy the advantages provided by deep learning (often high accuracy, high performance) for many useful and ubiquitous computing applications in their day to day lives. Finally, by doing continual learning on the device we will move one inch closer to machines that can reflect true human intelligence.