CAREER: Resilient and Efficient Automatic Control in Energy Infrastructure: An Expert-Guided Policy Optimization Framework

该学院早期职业发展（CAREER）奖支持将利用尖端人工智能技术的研究，以显着提高广泛的能源基础设施系统中自动控制系统的弹性和效率。这一举措至关重要，因为它解决了实践中使用的现有基于学习的决策框架所面临的若干重大限制。这项研究将通过开发一个分析严谨且实际可实施的框架来弥合这些关键的知识差距，该框架将强化学习与数学优化相结合，沿着专家在环指导。这项研究的成功应用预计将提高效率，稳定性和安全性，使能源基础设施能够快速，安全地应对不确定性和破坏性事件。将这项研究整合到华盛顿大学的课程中，将为研究生和本科生提供强化学习的培训和学习机会。教育和推广活动旨在通过各种举措提高K-12和大学生的认识和兴趣，包括互动式人工智能游戏培训平台，补充当地高中课堂课程的视频模块，该项目创造性地将分布鲁棒优化的原理应用于政策梯度强化学习方法，提高了在线策略样本的效率，并保持了稳定性。该模型的上级数值性能源于其无限制的策略分布，拒绝免费的政策更新，以及单调性能和全局收敛保证通过Wasserstein度量为基础的政策优化。专家在环强化学习框架有效地利用专家演示和反馈来确保系统安全运行，加速学习并增强算法收敛。通过修改“敏感”情况下的优势函数，该框架引导学习方向，并在有限样本的情况下解决了强化学习的弱点。本研究将回答三个关键问题：如何有效地利用专家反馈？如何确定需要专家干预的国家？ 如何实现独立于专家输入的最优和稳定的策略？这项工作产生的创新数学模型和算法将有助于解决在线决策的挑战，更好地运营和管理复杂的能源基础设施systems.This奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

This Faculty Early Career Development (CAREER) award supports research that will leverage cutting-edge artificial intelligence technologies to significantly enhance the resilience and efficiency of automated control systems within a broad class of energy infrastructure systems. This initiative is crucial as it addresses several substantial limitations faced by existing learning-based decision-making frameworks used in practice. This research will bridge these critical knowledge gaps by developing an analytically rigorous and practically implementable framework that integrates reinforcement learning with mathematical optimization, along with expert-in-the-loop guidance. The successful application of this research is anticipated to yield improvements in efficiency, stability, and security, empowering the energy infrastructure to respond rapidly and securely to uncertainty and disruptive events. Integration of this research into the curriculum at University of Washington will foster training and learning opportunities in reinforcement learning for both graduate and undergraduate students. Educational and outreach activities are designed to increase awareness and interest among K-12 and college students through diverse initiatives, including an interactive artificial intelligence game training platform, video modules to supplement classroom lessons for local high schools, and research engagement with underrepresented students.This project creatively applies the principles of distributionally robust optimization to policy gradient reinforcement learning methods for improving online policy sample efficiency and maintaining stability. The model’s superior numerical performance stems from its unrestricted policy distribution, rejection-free policy updates, as well as monotonic performance and global convergence guarantee through Wasserstein metric-based policy optimization. The expert-in-the-loop reinforcement learning framework effectively leverages expert demonstrations and feedback to ensure safe system operation, accelerate learning, and enhance algorithm convergence. By modifying the advantage function in "susceptible" situations, the framework guides learning direction and addresses reinforcement learning’s weaknesses with limited samples. This research will answer three key questions: How to effectively utilize expert feedback? How to identify states that require expert intervention? And how to achieve an optimal and stable policy independent of expert input? The innovative mathematical models and algorithms generated by this work will contribute to addressing online decision-making challenges for better operations and management of complex energy infrastructure systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.