EAGER: Real-Time: Reinforcement, Meta, and Episodic Learning for Control under Uncertainty

机器学习和人工智能是未来几十年最重要的通用技术之一，有可能改变社会的方方面面，从健康、制造、商业、教育到安全。在过去的十年中，在使用深度神经网络、创新的训练算法、包括专用硬件(图形处理器、张量处理单元)和大数据集的计算资源的推动下，机器学习取得了非常重要和令人印象深刻的进展。其中一些进展与神经科学对人类大脑如何学习实时做出决定的理解有关。然而，在工程系统的实时控制和决策中使用这些技术存在重大挑战，其中稳定性、可靠性和安全性是最重要的问题。该项目旨在将机器学习和神经科学的主要进展与控制系统联系起来，从而推动无数应用领域的发展。现代工程系统正变得越来越复杂。它们包括由(IoT)连接的设备、系统和人类/社会代理组成的大型异质分布式网络，例如交通、能源、水、制造、卫生和农业。一个主要的挑战是这些系统在大不确定性下的性能、稳定性和可靠性。其目标是扩大我们对学习和控制的理解和集成，以推导出用于各种工程应用的基于学习的控制系统开发的原理和算法。虽然强化学习和随机动态控制之间有重要的历史联系，但利用机器学习中正在进行的和未来的进展进行控制的潜力仍未得到充分开发。控制系统领域有深厚而坚实的理论和数学基础，具有线性、非线性、鲁棒、自适应、随机、分布式和模型预测控制系统的全面和成熟的框架。同样重要的是，控制系统在多个领域都有应用，如航空航天、汽车、制造、能源、交通、农业、水以及许多其他工程和社会技术系统。尽管有丰富的理论基础和重要的应用，但传统控制技术的适用范围仅限于基本系统的良好数学模型可用的情况，以及环境不确定性不太大的情况。这一探索性研究项目旨在通过系统和控制中的新问题公式来克服这些限制，这些问题的制定受到机器学习最新发展的新见解的启发。重点将放在受神经科学和强化学习启发的新型控制体系结构上。除了架构创新，该项目还将通过将快速(一次性)学习、元学习和情景控制的想法整合到控制算法中，来探索稳定性、性能和不确定性问题。来自这个项目的想法将成为加州大学欧文分校将教授的一门新的研究生水平的控制学习课程的核心。由此产生的课程材料将提供给研究社区，并将使全国感兴趣的研究生受益。此外，短期课程将在主要的专业会议上提供，例如，美国控制会议，IEEE决策与控制会议。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning and artificial intelligence are among the most important general purpose technologies for the coming decades, with potential to transform all aspects of society from health, to manufacturing, to business, to education, and to security. In the last decade, there have been very important and impressive advances in machine learning driven by the use of deep neural networks, innovative training algorithms, computational resources including specialized hardware (graphics processors, tensor processing units), and large datasets. Some of these developments have connections to emerging understanding from neuroscience on how the human brain learns to makes decisions in real-time. However, there are major challenges in the use of these techniques in real-time control and decision making for engineering systems where stability, reliability, and safety are paramount concerns. This project aims to connect major advances in machine learning and neuroscience to control systems and thereby advance myriad application domains. Modern engineered systems are increasingly complicated. They comprise large heterogeneous distributed networks of (IoT) connected devices, systems, and human/social agents, e.g., transportation, energy, water, manufacturing, health and agriculture. A major challenge is performance, stability and reliability of these systems under large uncertainties. The goal is to expand our understanding and integration of learning and control to derive principles and algorithms for the development of learning-based control systems for a variety of engineering applications. While there are significant historical connections between reinforcement learning and stochastic dynamic control, the potential for leveraging ongoing and future advances in machine learning for control remains significantly under- explored. The field of control systems has deep and solid theoretical and mathematical foundations with comprehensive and well-established frameworks for linear, nonlinear, robust, adaptive, stochastic, distributed, and model-predictive control systems. Equally importantly, control systems have applications in multiple domains, such as aerospace, automotive, manufacturing, energy, transportation, agriculture, water, and many other engineered and socio-technical systems. Despite this rich spectrum of theoretical foundations and important applications, the domain of applicability of traditional control techniques is limited to situations where good mathematical models of the underlying systems are available, and where the environmental uncertainty is not too large. This exploratory research project is aimed at overcoming these limitations via novel problem formulations in systems and control inspired by new insights coming from recent developments in machine learning. A key focus will be on novel control architectures inspired by neuroscience and reinforcement learning. Besides architectural innovations, the project will explore questions of stability, performance, and uncertainty by integrating ideas from rapid (one-shot) learning, meta-learning, and episodic control into control algorithms. The ideas from this project will be at the core of a new graduate level course in learning for control which will be taught at the University of California, Irvine. The resulting course materials will be made available to the research community and will benefit interested graduate students across the nation. In addition, short courses will be offered at major professional conferences, e. g., American Control Conference, IEEE Conference on Decision and Control.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.