CAREER: Towards Provenance-Driven Understanding of Machine Learning Robustness

点击翻译按钮获取中文摘要

Machine Learning (ML) is increasingly used in socially critical applications such as self-driving cars, medicine, finance, and criminal justice. However, ML is also susceptible to adversaries who can attack both the data models are trained on and the ML models themselves. This can lead to poor behavior in the models and poor decisions in the people who use them. This project’s goal is to advance our ability to detect and respond to attacks through focusing on provenance: systematic capture of the data and training methods used in building models, along with the inference processes and decisions made after they are deployed. By capturing these data and developing methods to use the data when assessing risks, auditing models, and forensically analyzing incidents, the work will make ML systems both more robust and more accountable around attacks. These capabilities will in turn benefit organizations that develop and use ML models, along with policymakers and regulators who oversee their effects. The work is organized into three main thrusts. The first thrust focuses on systematic capture and characterization of pre-deployment (training) and post-deployment (inference) provenance, focusing on what constitutes training and inference provenance and the innate nondeterminism of ML computations. In particular, training and inference metadata, training progression, inference computation dynamics, and per-label characterization approaches will be explored. The second thrust will use these data for provenance-driven detection of training data poisoning and model evasion across a range of threat models and application domains. For poisoning detection, both similarity-based and distribution shift detection-based approaches will be pursued while for evasion detection, inference provenance will be analyzed empirically and structurally. The third thrust focuses on developing post-compromise forensics capabilities with the goal of tracing back attacks to their cause(s) and mitigating future attacks. Integrated with these three thrusts is an educational plan that includes developing new courses on ML trustworthiness for undergraduate and graduate students, robust ML-focused ethical hacking competitions for undergraduates, and K-12 summer camps on robust ML to develop and diversify the next generation of cybersecurity workers.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.