MultiTasking and Continual Learning for Audio Sensing Tasks on Resource-Constrained Platforms

资源受限平台上音频传感任务的多任务处理和持续学习

• 批准号: EP/X01200X/1
• 负责人: Jagmohan Chauhan
• 金额: $ 58.18万
• 依托单位: University of Southampton
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FX01200X%2F1
• 关键词: MultiTasking; Continual; Learning; Audio; Sensing

Deep learning (a form of machine learning) has been highly successful in solving many complex tasks in the domain of computer vision, natural language processing and speech processing due to its ability to learn effective representations from raw data. This success is pushing demands to deploy deep learning models on more resource-constrained computing platforms such as Micro Controller Units (MCUs) as they are highly accurate and deploying them locally on the device will give enhanced privacy to user data. Proliferation of cheap sensors and IoT (Internet of Things) devices will further fuel this demand and the recent trends in ever growing field of TinyML (putting machine/deep learning on tiny devices) is an indication thereof. Recently, significant gains have been made in realizing the goal of deploying deep learning models efficiently on resource-constrained devices. However, the focus is still limited to solving single tasks efficiently. Additionally, the models are static; they cannot learn with time. We think it is time to go beyond the static "once learnt and deploy" deep learning models. The ability to multi-task and learn continuously is required to adapt to unseen changes, learn new information, and handle multiple disparate applications. However, accommodating these abilities on resource-constrained devices is extremely challenging: limited memory and compute power. To this end, this project aims to develop a range of techniques that make deep learning models learn on the fly and solve multiple tasks efficiently (low latency, low power) on resource-constrained devices. Overall, the project goals are to: (a) design an optimal memory management scheme to keep multiple deep learning models in the available memory of the device, (b) devise novel scheduling strategies that can distribute the workload on available processing cores on device efficiently for an application and execute tasks in parallel, and finally (c) come up with a method that will combine continual learning with few shot learning paradigm to allow models to learn continuously with few annotated data on device. The developed techniques will be tested on an embedded audio platform for a variety of audio sensing tasks such as keyword spotting, audio scene analysis (localization, scene classification, sound classification), and speech enhancement. The focus on audio is due to its rising prominence in many core applications, such as home hubs like Alexa, ecological monitoring, disease diagnostics, preventive maintenance, hearables and accessibility devices such as hearing aids. The resulting innovations from this work will have numerous benefits. First of all, developing efficient deep learning solutions that can run on-device would save power leading to a lower digital carbon footprint, a far more sustainable society and contributing to the UK's mission of NetZero 2050. Local execution further leads to enhanced user privacy as data never leaves the device. This is also a key benefit at places where network communication is absent or can be expensive such as LMIC (Low- and middle-income countries). Secondly, creating such solutions means users can enjoy the advantages provided by deep learning (often high accuracy, high performance) for many useful and ubiquitous computing applications in their day to day lives. Finally, by doing continual learning on the device we will move one inch closer to machines that can reflect true human intelligence.

深度学习（机器学习的一种形式）在解决计算机视觉、自然语言处理和语音处理领域的许多复杂任务方面非常成功，因为它能够从原始数据中学习有效的表示。这一成功推动了在资源受限的计算平台（如微控制器单元（MCU））上部署深度学习模型的需求，因为它们高度准确，并且在设备上本地部署它们将增强用户数据的隐私性。廉价传感器和物联网（IoT）设备的激增将进一步推动这一需求，最近不断增长的TinyML（将机器/深度学习放在微型设备上）领域的趋势就是一个迹象。最近，在实现在资源受限的设备上有效部署深度学习模型的目标方面取得了重大进展。然而，重点仍然局限于有效地解决单个任务。此外，模型是静态的;它们不能随着时间的推移而学习。我们认为是时候超越静态的“一旦学习和部署”的深度学习模型了。多任务和持续学习的能力是适应不可见的变化、学习新信息和处理多个不同应用程序所必需的。然而，在资源受限的设备上提供这些功能是非常具有挑战性的：有限的内存和计算能力。为此，该项目旨在开发一系列技术，使深度学习模型能够在资源受限的设备上实时学习并有效地解决多个任务（低延迟，低功耗）。总体而言，项目目标是：（a）设计最佳存储器管理方案以将多个深度学习模型保持在设备的可用存储器中，（B）设计新颖的调度策略，其可以针对应用有效地将工作负载分布在设备上的可用处理核上并且并行地执行任务，以及最后（c）提出将联合收割机连续学习与少镜头学习范例相结合的方法，以允许模型在设备上具有很少注释数据的情况下连续学习。开发的技术将在嵌入式音频平台上进行测试，用于各种音频传感任务，如关键字定位，音频场景分析（本地化，场景分类，声音分类）和语音增强。对音频的关注是由于其在许多核心应用中的日益突出，例如Alexa等家庭集线器，生态监测，疾病诊断，预防性维护，助听器和助听器等无障碍设备。 这项工作产生的创新将带来许多好处。首先，开发可以在设备上运行的高效深度学习解决方案将节省电力，从而降低数字碳足迹，建立一个更具可持续性的社会，并为英国的NetZero 2050使命做出贡献。本地执行进一步增强了用户隐私，因为数据永远不会离开设备。这也是在网络通信不存在或可能昂贵的地方，如LMIC（低收入和中等收入国家）的一个关键优势。其次，创建这样的解决方案意味着用户可以在日常生活中享受深度学习为许多有用和无处不在的计算应用提供的优势（通常是高准确性，高性能）。最后，通过在设备上进行持续学习，我们将向能够反映真正人类智能的机器更近一步。