CAREER: Towards a Principled Framework for Resilient, Data Efficient and Scalable Reinforcement Learning for Control

传统控制系统设计的成功关键取决于具有已知参数、充分理解的不确定性来源和明确指定的目标的易处理系统模型的可用性。这些假设在智能和自主的大规模工程系统（如下一代电力系统）的新兴范式中不再成立。利用来自此类系统的大量数据的数据驱动机器学习方法可以提供一条有前途的前进道路。然而，与机器学习在图像识别等分类问题上取得的巨大成功不同，解决“学习控制”问题的强化学习（RL）的成就仅限于更结构化或模拟的环境，其在现实世界工程系统中的成功并不突出。有三个关键问题严重阻碍了RL在现实世界工程系统中的成功：缺乏弹性，数据效率和可扩展性。该CAREER提案通过克服弹性，数据效率和可扩展性的基本挑战，为大型现实世界工程系统的控制算法的基于RL的设计开发了一种原则性方法。感兴趣的主要应用领域是电力系统，它指导问题的制定和解决方法，并使用真实世界的例子来信任算法。该项目有一个创新的教育计划，其中包括“Aggie DeepRacer项目”，该项目遵循“体验式学习”的方法，将RL的研究融入教育课程。通过与路易斯·斯托克斯少数民族参与联盟（LSAMP）计划合作，指导来自代表性不足的少数民族的学生，并接待来自少数民族学生比例高的低社会经济地位学校的教师，加强了该项目。项目成果包括为高中生开发基于活动的学习模块，并在Aggie STEM夏令营和物理与工程节上展示。通过开发数据驱动和基于学习的方法来有效控制电力系统，该项目还有助于降低燃料和运营成本，从而显著提高整个能源系统的可靠性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The success of the traditional control system design depends crucially on the availability of tractable system models with known parameters, well-understood sources of uncertainty and clearly specified objectives. These assumptions are no longer true in the emerging paradigm of intelligent and autonomous large-scale engineering systems, such as next generation electricity systems. A data-driven machine learning approach that takes advantage of large amounts of data coming from such systems can provide a promising path forward. However, unlike the remarkable successes of machine learning in classification problems such as image recognition, reinforcement learning (RL) that addresses the problem of “learning to control” has seen achievements limited to more structured or simulated environments, and its successes in real-world engineering systems are not as prominent. There are three critical issues that significantly impede the success of RL in real-world engineering systems: lack of resiliency, data efficiency, and scalability. This CAREER proposal develops a principled approach for the RL-based design of control algorithms for large-scale real-world engineering systems, by overcoming the fundamental challenges of resiliency, data efficiency, and scalability. The main application domain of interest is electricity systems, which guides the problem formulation and solution approaches, and lends credence to the algorithms using real-world examples. The project has an innovative education plan that includes the ‘Aggie DeepRacer Project’ that follows an ‘experiential learning’ approach for integrating the research in RL into the educational curriculum. Mentoring students from underrepresented minorities through collaboration with the Louis Stokes Alliances for Minority Participation (LSAMP) program, and hosting teachers from low socioeconomic status schools with a high percentage of minority students strengthens the project. Project outcomes include development of activity-based learning modules for high school students and presenting them at the Aggie STEM summer camp and Physics and Engineering Festival. By developing a data-driven and learning-based approach for efficient control of power systems, this project also contributes to reducing the cost of fuel and operations, and hence significantly increasing the reliability of the overall energy system.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.