RI: Small: Reinforcement Learning with Predictive State Representations

像动物和人类一样，能够预测其行为的短期和长期后果的人工自主代理随后可以计划自己的行为，采取更智能的行动，并获得更大的回报。与依赖预建模型的代理相比，能够从经验中学习此类预测模型的代理可以在其智能方面更加稳健。PI和研究生专注于特别具有挑战性但自然的情况，在这种情况下，来自特工传感器的遥远过去的观察可能会继续影响对未来很长一段时间行动后果的预测。(例如，观察你早上把车停在哪里将有助于预测当天晚些时候你会在哪里看到车。)在这种“部分可观察”的环境中，学习预测模型的方法有两大类。有限历史模型使用短期观察历史来预测以行动为条件的未来观察；这些模型学习起来很快，但有限，因为它们不能捕捉长期历史的影响。潜变量模型可以通过设置隐变量或潜变量来捕捉长期历史的影响，这些变量可以捕捉环境的真实状态(例如，汽车的位置)，但这种模型很难学习，因为潜在变量必须从数据中推断出来。这个项目建立在PI和其他人之前在第三种方法上的工作基础上，称为预测性状态表示(PSR)，在这种方法中，代理维护对未来观察的预测，条件是未来的行动作为历史的汇总表示；这些模型既可以快速学习，也可以捕获长期历史的影响。该项目为层次化模型、基于丰富特征的模型以及局部和模块化模型开发了基于PSR的新方法和算法。该项目将新方法应用于主动感知和机器人技术中具有挑战性的应用。此外，还将发展对这些更丰富、更新的方法的理论理解。总而言之，该项目极大地扩展了PSR方法及其理论基础和算法的适用范围。更广泛的影响：允许人工代理强大地建立预测模型的新方法将促进人工智能、强化学习、控制、运筹学、心理学和神经科学等领域的知识状态。PI正在共同领导一项努力，在密歇根大学创建一个新的数据科学本科学位，由计算机科学与工程与统计联合管理。这个未来的学位以及其他目前的本科生研究项目将有针对性地为这个项目招收、指导和培训学生。

Like animals and humans, artificial autonomous agents that are able to predict short-term and long-term consequences of their actions can then plan their behavior, act more intelligently, and achieve greater reward. Agents that can learn such predictive models from experience can be more robust in their intelligence than agents that rely on pre-built models. The PI and graduate students are focused on the particularly challenging but natural case where observations from the agent's sensors far in the past can continue to influence the predictions of consequences of actions long into the future. (For example, the observation of where you park the car in the morning will help predict where you will see the car later in the day.) There are two broad classes of approaches to learning predictive models in such 'partially observable' settings. Finite-history models use short-term history of observations to predict future observations conditioned on actions; these are fast to learn but are limited because they cannot capture the effects of long-term history. Latent-variable models can capture the effects of long-term history by positing hidden or latent variables that capture the true state of the environment (e.g., the location of the car), but such models are difficult to learn because the latent variables have to be inferred from data. This project builds on previous work by the PI and others on a third approach, called Predictive State Representations (or PSRs), in which the agent maintains predictions of future observations conditioned on future actions as a summary-representation of history; these models can both be fast to learn and capture the effect of long-term history. This project develops new PSR-based methods and algorithms for hierarchical models, rich-feature-based models, and local and modular models. The project applies the new methods to challenging applications from active perception and robotics. In addition, theoretical understanding of these richer and newer methods will be developed. Altogether the project significantly expands the applicability of PSR-methods as well as their theoretical foundations and algorithms. Broader Impacts: New methods that allow artificial agents to robustly build predictive models would advance the state of knowledge across the fields of artificial intelligence, reinforcement learning, control, operations research, psychology, and neuroscience. The PI is co-leading an effort to create a new undergraduate degree in Data Sciences at the University of Michigan to be jointly managed by Computer Science & Engineering and Statistics. This future degree as well as other current undergraduate research programs will be targeted to recruit, mentor, and train students for this project.