EAGER: Real-Time: Precision Reserves from Flexible Loads: An Online Reinforcement Learning Approach

本提案探索了一种在线强化学习框架，该框架可以提供高容量评级和调度许多最终用户级灵活资源（如游泳池）。与传统的静态和统一处理小容量额定的最终用户负载并通过启发式算法调度它们的方法形成鲜明对比，所提出的框架将为学习最终用户负载的未知参数并自适应控制它们提供理论上严格和实际可扩展的方法。智力优势：(i)该提案将说明通过可扩展的实时估计和控制，而不是传统的基于启发式的调度算法，在提供旋转储备时，最终用户需求响应大幅增加容量信贷的可能性。（ii）本提案将引入一种使用在线强化学习框架的学习和自适应控制算法，以解决消费者特定参数未知时的操作问题。（iii）本提案将引入一种基于索引的学习和调度算法，该算法仅随最终用户数量线性扩展。（iv）该方案将测试一种数据驱动的最优调度，该调度共同使聚合器的利润最大化，并从少量灵活用户的集合中跟踪所需的储备供应轨迹。本文的研究可推广到交通、通信等工程动态系统中出现的许多具有不确定性的资源调度问题。更广泛的影响：一旦成功，该项目将提供一种系统的方法，以更低的成本从灵活的最终用户资源中以可靠和环境可持续的方式获得旋转储备。该团队将为200多名目前就读于德克萨斯农工大学相关领域课程的本科生和研究生介绍新的课程模块，主题是动态系统中数据驱动的在线学习，它将强化学习、动态控制和优化紧密结合在一起。这个团队将继续保持吸引本科生参与研究的良好记录，特别是那些代表性不足的群体。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This proposal explores an online reinforcement learning framework that can provide high capacity rating and scheduling of many end user-level flexible resources such as swimming pools. In sharp contrast with conventional approaches of statically and uniformly treating end user loads with small capacity rating and scheduling them via heuristics based algorithms, the proposed framework will provide a theoretically rigorous and practically scalable approach for learning the unknown parameters of end user loads and adaptively controlling them with provable guarantees.Intellectual Merit: (i) This proposal will illustrate the possibility of substantial increasing of capacity credit from end user demand response in provision of spinning reserves via scalable real-time estimation and control as opposed to the conventional heuristic based scheduling algorithms. (ii) This proposal will introduce a learning and adaptive control algorithm using the framework of online reinforcement learning to address the operational problems when the consumer specific parameters are unknown. (iii) This proposal will introduce an index-based learning and scheduling algorithm that scales only linearly with the number of end users. (iv) This proposal will test a data-driven optimal scheduling that jointly maximize the profit for the aggregator and track the required reserve provision trajectory from the collection of even a small number of flexible users. The proposed research is generalizable towards many resource scheduling problems with uncertainty that arise in the context of transportation, communication, and other engineering dynamical systems.Broader Impacts:Once successful, this project will provide a systematic approach for obtaining spinning reserve at muchless cost from flexible end user resources in a provably reliable and environmentally sustainable way.This team will introduce new course modules on the topic of data-driven online learning in dynamical systems, which closely integrates reinforcement learning, dynamical control, and optimization for more than 200 undergraduate and graduate students currently enrolled in related areas courses at Texas A&M. This team will continue the strong track record of engaging undergraduate students for research, in particular the under-representative groups.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.