A Comprehensive Study Towards Lifelong Federated Learning Systems

增加输入数据大小可以显著减少许多人工智能（AI）算法的学习错误。通过利用分布式和不断增长的数据（即，来自智能手机、可穿戴设备和不同机构的数据）。然而，传统的人工智能方法需要将数据集中在一个设备或数据中心上，由于内存和隐私问题（即，在智能手机上收集的大量键盘数据）。因此，我的研究计划旨在推进人工智能从不断增长的分布式数据中学习。我将专注于联邦学习（FL），这是一种新兴技术，允许公司，研究人员和个人在不集中汇总原始数据的情况下协作做出明智的决策。为了提高FL模型在随着时间的推移收集的越来越多的数据上的性能，我将FL与持续学习（CL）结合起来，这是一种有效的学习策略，可以从数据流中不断学习新知识，而无需从头开始重新训练模型。这种新的融合FL和CL策略被称为持续FL。持续FL旨在解决以下关键挑战：1）使FL模型适应具有变化外观或标签分布的新客户端; 2）教导FL模型在新任务上运行而不忘记过去的任务; 3）防御恶意新客户端的威胁。为此，我将首先开发新的对齐方法，在隐私约束下协调新老客户端之间的不同数据模式（目标1）。其次，为了防止FL模型在学习新任务时忘记旧知识，我将提出两种新的方法，模型手术和无记忆回复，有效地将旧知识转移到新一轮的FL模型训练中（目标2）。最后，考虑到持续FL中的安全问题，即新客户端可能是恶意的，我将开发恶意客户端检测方法和鲁棒的聚合策略，以提高FL系统的鲁棒性（目标3）。这项研究的意义在于它的新奇性的算法和实践部署和维护FL模型的非平稳，时移和分布式数据流。而不是不同的FL系统为不同的数据和任务，连续FL将使FL系统不断学习新的知识后，模型建立，并逐步获得处理各种任务的能力，从而实现终身FL系统。我的研究将有助于使这一愿景更接近现实，并提供一个使用FL推进人工智能开发的机会。因此，它使人工智能能够应用于更广泛的现实世界条件和应用程序，其中数据是孤立的和敏感的，如医疗保健，金融和犯罪检测。该计划将培养3名博士，1名硕士和3名大学生，为他们提供增强的研究技能，并为他们在学术界和工业界的职业机会做好准备。

Increasing the input data size can significantly reduce the learning error for many artificial intelligence (AI) algorithms. The demand has increased imperatively for improving the accuracy of AI-based automatic decision-making by utilizing distributed and growing data (i.e., data from smartphones, wearable devices, and different institutions). However, traditional AI methods require centralizing data on one device or data center, which is not always possible due to memory and privacy concerns (i.e., numerous keyboards data collected on smartphones). Therefore, my research program aims to advance AI on learning from growing distributed data. I will focus on federated learning (FL), an emerging technique allowing companies, researchers, and individuals collaboratively make informed decisions without aggregating raw data centrally. To improve FL models' performance on the growing amount of data collected over time, I will incorporate FL with continuous learning (CL), an efficient learning strategy that continuously learns new knowledge from a stream of data without retraining models from scratch. This novel integrated FL and CL strategy is called continual FL. Continual FL aims to tackle the following key challenges: 1) adapt FL model to new clients with changing appearance or label distributions; 2) teach FL model to function on new tasks without forgetting past tasks, and 3) defend against the threat of malicious new clients. To this end, I will first develop new alignment methods that harmonize different data patterns between new and old clients under privacy constraints (Obj 1). Second, to prevent FL model from forgetting old knowledge when learning new tasks, I will propose two novel approaches, model surgery and memory-free reply, that efficiently transfer the old knowledge into the new round of FL model training (Obj 2). Lastly, considering a security issue in continual FL that new clients may be malicious, I will develop malicious client detection methods and robust aggregation strategies to improve the robustness of FL system (Obj 3). The significance of this research lies in its novelty on both algorithms and the practice of deploying and maintaining FL models for nonstationary, time-shifting, and distributed data streams. Instead of varying FL systems for different data and tasks, continual FL will enable an FL system to endlessly learn new knowledge after the model is built and gradually gain capacity to handle various tasks, thus achieving lifelong FL systems. My research will help make this vision one step closer to reality and provide an opportunity of advancing AI development using FL. Thus, it enables AI to be applied to a wider range of real-world conditions and applications where data is isolated and sensitive, such as healthcare, finance, and crime detection. This program will train 3 PhD, 1 MSc and 3 UGs, equipping them with enhanced research skills and preparing them for career opportunities in both academia and industry.