EAGER: Real-Time: Precision Reserves from Flexible Loads: An Online Reinforcement Learning Approach

该提案探索了一种在线强化学习框架，可以提供高容量评级和许多最终用户级灵活资源（如游泳池）的调度。与传统的静态和统一处理具有小容量额定值的终端用户负载并通过基于算法的调度方法形成鲜明对比，所提出的框架将提供一种理论上严格且实际上可扩展的方法，用于学习终端用户负载的未知参数并以可证明的保证自适应地控制它们。（i）该建议将说明通过可扩展的实时估计和控制而不是传统的基于启发式的调度算法，在提供旋转储备时，从最终用户需求响应大幅增加容量信用的可能性。(ii)该建议将引入一种学习和自适应控制算法，使用在线强化学习的框架来解决消费者特定参数未知时的操作问题。(iii)该提案将引入一种基于索引的学习和调度算法，该算法仅随最终用户的数量线性扩展。(iv)该提案将测试数据驱动的最佳调度，该调度联合最大化聚合器的利润，并跟踪即使是少量灵活用户的集合所需的准备金提供轨迹。 所提出的研究可推广到许多资源调度问题的不确定性，出现在运输，通信和其他工程动态系统的背景下。一旦成功，该项目将提供一种系统化的方法，以可证明可靠和环境可持续的方式，从灵活的最终用户资源中以低成本获得旋转储备。该团队将引入关于动态系统中数据驱动在线学习主题的新课程模块，该模块将强化学习，动态控制，和优化的200多名本科生和研究生目前就读于相关领域的课程在得克萨斯州A& M。该团队将继续保持吸引本科生参与研究的良好记录，特别是代表性不足的群体。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

This proposal explores an online reinforcement learning framework that can provide high capacity rating and scheduling of many end user-level flexible resources such as swimming pools. In sharp contrast with conventional approaches of statically and uniformly treating end user loads with small capacity rating and scheduling them via heuristics based algorithms, the proposed framework will provide a theoretically rigorous and practically scalable approach for learning the unknown parameters of end user loads and adaptively controlling them with provable guarantees.Intellectual Merit: (i) This proposal will illustrate the possibility of substantial increasing of capacity credit from end user demand response in provision of spinning reserves via scalable real-time estimation and control as opposed to the conventional heuristic based scheduling algorithms. (ii) This proposal will introduce a learning and adaptive control algorithm using the framework of online reinforcement learning to address the operational problems when the consumer specific parameters are unknown. (iii) This proposal will introduce an index-based learning and scheduling algorithm that scales only linearly with the number of end users. (iv) This proposal will test a data-driven optimal scheduling that jointly maximize the profit for the aggregator and track the required reserve provision trajectory from the collection of even a small number of flexible users. The proposed research is generalizable towards many resource scheduling problems with uncertainty that arise in the context of transportation, communication, and other engineering dynamical systems.Broader Impacts:Once successful, this project will provide a systematic approach for obtaining spinning reserve at muchless cost from flexible end user resources in a provably reliable and environmentally sustainable way.This team will introduce new course modules on the topic of data-driven online learning in dynamical systems, which closely integrates reinforcement learning, dynamical control, and optimization for more than 200 undergraduate and graduate students currently enrolled in related areas courses at Texas A&M. This team will continue the strong track record of engaging undergraduate students for research, in particular the under-representative groups.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.