A Comprehensive Study Towards Lifelong Federated Learning Systems

点击翻译按钮获取中文摘要

Increasing the input data size can significantly reduce the learning error for many artificial intelligence (AI) algorithms. The demand has increased imperatively for improving the accuracy of AI-based automatic decision-making by utilizing distributed and growing data (i.e., data from smartphones, wearable devices, and different institutions). However, traditional AI methods require centralizing data on one device or data center, which is not always possible due to memory and privacy concerns (i.e., numerous keyboards data collected on smartphones). Therefore, my research program aims to advance AI on learning from growing distributed data. I will focus on federated learning (FL), an emerging technique allowing companies, researchers, and individuals collaboratively make informed decisions without aggregating raw data centrally. To improve FL models' performance on the growing amount of data collected over time, I will incorporate FL with continuous learning (CL), an efficient learning strategy that continuously learns new knowledge from a stream of data without retraining models from scratch. This novel integrated FL and CL strategy is called continual FL. Continual FL aims to tackle the following key challenges: 1) adapt FL model to new clients with changing appearance or label distributions; 2) teach FL model to function on new tasks without forgetting past tasks, and 3) defend against the threat of malicious new clients. To this end, I will first develop new alignment methods that harmonize different data patterns between new and old clients under privacy constraints (Obj 1). Second, to prevent FL model from forgetting old knowledge when learning new tasks, I will propose two novel approaches, model surgery and memory-free reply, that efficiently transfer the old knowledge into the new round of FL model training (Obj 2). Lastly, considering a security issue in continual FL that new clients may be malicious, I will develop malicious client detection methods and robust aggregation strategies to improve the robustness of FL system (Obj 3). The significance of this research lies in its novelty on both algorithms and the practice of deploying and maintaining FL models for nonstationary, time-shifting, and distributed data streams. Instead of varying FL systems for different data and tasks, continual FL will enable an FL system to endlessly learn new knowledge after the model is built and gradually gain capacity to handle various tasks, thus achieving lifelong FL systems. My research will help make this vision one step closer to reality and provide an opportunity of advancing AI development using FL. Thus, it enables AI to be applied to a wider range of real-world conditions and applications where data is isolated and sensitive, such as healthcare, finance, and crime detection. This program will train 3 PhD, 1 MSc and 3 UGs, equipping them with enhanced research skills and preparing them for career opportunities in both academia and industry.