Biologically-Inspired Robust Adaptive Dynamic Programming for Continuous-Time Stochastic Systems

该项目旨在开发受神经生物学启发的工具和方法，以满足建立类似大脑的强化学习系统的需求，并最终促进对大脑功能的理解。该项目寻求从根本上解决受随机效应、非线性和动态不确定性影响的大型复杂系统的稳健优化管理所产生的具有挑战性的问题。该项目的研究成果将为智能电网、机器人和智能交通系统等新兴工程应用贡献新的解决方案。拟议的研究将对该机构的教育产生重大的直接影响。该项目的跨学科性质应该吸引来自几个系的学生。该项目团队将致力于自适应动态规划(ADP)的随机变体，用于受随机和动态扰动的连续时间系统。对于复杂系统的最优控制设计，ADP是一种实际可行的数据驱动、非基于模型的方法。ADP已被广泛用于马尔可夫决策过程，主要集中在离散和有限状态空间，以及确定性(离散和连续时间)动态系统。在存在动态不确定性的情况下，稳定性和鲁棒性问题很少被系统地解决。对于涉及复杂的现代工程系统或生物系统的问题，稳定性是一个重要的问题，直接应用现有的ADP结果似乎不具成效，甚至不太可能成功。因此，有必要开发新的工具和方法，在连续时间和连续状态空间中对一般随机系统进行ADP设计，并进行严格的稳定性分析和收敛分析。该研究的创新之处在于对强化学习、随机系统理论和非线性控制理论等技术的应用和扩展。该提案的具体目标是开发用于线性和非线性随机系统的随机自适应动态规划的工具和方法，具有对动态不确定性的鲁棒性的随机自适应最优控制，以及应用于人类电机系统。将进行严格的稳定性证明、学习算法的收敛分析和稳健性分析。我们将研究一类重要的具有乘性和加性噪声的线性和非线性连续时间模型，以及基于非模型的随机最优控制器设计。除了工程应用，人们认为，将ADP和计算神经科学的研究结合在一起，可能会产生新的方法来诊断和治疗影响肌肉协调的神经退行性遗传疾病。帕金森氏症就是一种这样的疾病，全球约有700万人受到影响，美国有100万人。将PI最近在稳健自适应动态规划的线性随机变量中的工作推广到人类运动控制的潜在的新的计算机制。

The project aims to develop tools and methods, inspired by neurobiology, for addressing the need in building brain-like reinforcement learning systems and, ultimately, contributing to the understanding of brain functions. The project seeks to address fundamentally challenging issues arising from the robust optimal management of large complex systems subject to stochastic effects, nonlinearity, and dynamic uncertainties. Research findings from this project will contribute new solutions to emerging engineering applications such as the smart electricity grid, robotics, and intelligent transportation systems. The proposed research will have a substantial direct impact upon education at the PI's institution. The interdisciplinary nature of the project should appeal to students from several departments.The project team will work on stochastic variants of adaptive dynamic programming (ADP) for continuous-time systems subject to stochastic and dynamic disturbances. ADP is a practically sound data-driven, non-model based approach for optimal control design in complex systems. ADP has been extensively studied for Markov decision processes, focusing mostly on discrete and finite state-space, and for deterministic (discrete- and continuous-time) dynamic systems. Stability and robustness issues in the presence of dynamic uncertainties are seldom addressed systematically. For problems involving complex modern engineering systems or biological systems, for which stability is an important concern, straightforward application of the existing ADP results does not seem productive or even likely to be successful. Hence, it is necessary to develop novel tools and methods for ADP design of general stochastic systems in continuous-time and continuous state-space, with rigorous stability and convergence analysis. The novelty of the proposed research consists of application and extension of techniques from reinforcement learning, stochastic systems theory, and nonlinear control theory. The specific goals of the proposal are the development of tools and methods for stochastic adaptive dynamic programming for linear and nonlinear stochastic systems, stochastic adaptive optimal control with robustness to dynamic uncertainties, and application to human motor systems. Rigorous stability proofs, convergence analysis of learning algorithms, and robustness analysis will be pursued. Important classes of continuous-time linear and nonlinear models with multiplicative and additive noise will be studied, along with non-model based, stochastic optimal controller designs. Beyond engineering applications, it is believed that bringing together ADP and research in computational neuroscience may yield new methodologies for the diagnosis and treatment of neurodegenerative genetic disorders that affect muscle coordination. One such medical condition is Parkinson's disease, which affects approximately seven million people globally, and one million in the United States. Generalizing the PI's recent work in linear stochastic variants of robust adaptive dynamic programming can lead to a potentially new computational mechanism for human motor control.