Collaborative Research: RI: Small: Advancing Theory and Practice of Trustworthy Machine Learning via Bi-Level Optimization

深度学习（DL）由于其上级预测能力而取得了显著的成功，在计算机视觉和自然语言处理方面有着广泛的应用。然而，它的一个关键缺点是缺乏可信度。也就是说，它们在训练过程中经常被过度烹饪，使得（1）学习的模型在测试时非常容易受到小输入扰动的影响（即缺乏鲁棒性）;以及（2）嵌入在训练数据中的有偏见的伪像可以被记忆，然后传递到决策过程（即缺乏公平性）。为了解决这些问题，该项目试图开发一个新的值得信赖的学习算法家族，该算法具有算法的通用性，理论上的可靠性以及对大规模数据集和模型的可扩展性。该项目的成果可以创建一个新的值得信赖的深度学习优化基础，不仅可以将鲁棒性和公平性纳入一个连贯的学习范式，还可以将深度学习的适用性扩展到一系列高风险应用，如自动驾驶和网络安全。计算机科学，应用数学和工程学的跨学科培训将提供给所有级别的学生，特别是来自代表性不足群体的学生。该项目的主要技术目标是通过双层优化（BLO）的透镜，即涉及两个嵌套优化任务的分层学习，推进可信DL的理论理解和实际实现。研究计划包括三个重点。第一个推力开发了一个新的面向BLO的鲁棒学习框架，包括对抗性实例和分布变化的防御。所开发的技术也适用于建立一个全栈（从训练时间到测试时间）的鲁棒性评估管道。第二个推力扩展了第一个推力，并开发了BLO算法，以共同提高两个实际场景中的鲁棒性和公平性，即在没有敏感属性注释的情况下学习，以及在缺乏训练数据和模型信息的情况下学习。第三个重点是为BLO开发可扩展和理论基础的计算方法，以实现高精度，高弹性和高吞吐量的值得信赖的学习范式。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning (DL) has achieved remarkable success owing to its superior prediction ability, with a wide range of applications in computer vision and natural language processing. Yet, one of its critical shortcomings is the lack of trustworthiness. That is, they are often overcooked during training such that (1) the learned model is highly vulnerable to small input perturbations at the testing time (namely, lack of robustness); And (2) biased artifacts embedded in the training data can be memorized and then passed on to the decision making process (namely, lack of fairness). To address these issues, this project attempts to develop a new family of trustworthy learning algorithms with algorithmic generality, theoretical soundness, and scalability to large-scale datasets and models. The outcome of this project could create a new optimization foundation of trustworthy DL that can not only unit robustness and fairness into one coherent learning paradigm but also expand the applicability of DL to a series of high-stakes applications such as autonomous driving and cybersecurity. Interdisciplinary training in computer science, applied mathematics, and engineering will be provided to all-level students, especially for students from underrepresented groups. The main technical aim of this project is to advance the theoretical understanding and practical implementations of trustworthy DL through the lens of bi-level optimization (BLO), namely, hierarchical learning involving two nested optimization tasks. The research plan consists of three thrusts. The first thrust develops a new BLO-oriented robust learning framework including defenses against adversarial instances and distribution shifts. The developed technique is also applied to building a full-stack (from train time to test time) robustness evaluation pipeline. The second thrust expands the first one and develops BLO algorithms to co-improve robustness and fairness in two practical scenarios, learning without sensitive attribute annotation, and learning with scarce training data and model information. The third thrust focuses on developing scalable and theoretically-grounded computational methods for BLO so as to achieve a high-accuracy, high-resilience, and high-throughput trustworthy learning paradigm. The project will result in the dissemination of shared toolbox and benchmarks to the broader optimization and machine learning communities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.