Memory-Based Operant Learning

PI将开发一种认知上合理的强化学习（RL）架构，作为动物和机器人工具学习的模型。虽然强化学习最初是受到动物学习现象的启发，但该领域的发展主要是通过解决人工智能问题。当前RL架构作为认知理论的一个主要限制是状态空间的表示。维持显式状态表示的模型（如Q~表）仅限于只有几个变量的简单域，而隐式表示状态空间的模型（例如，使用神经网络函数逼近器）需要大量的训练数据和与真实动物相比不合理的长时间训练。PI的方法是开发适合动物行为建模的状态空间的专门表示，并且可以支持所需的泛化。模拟动物的工作记忆将编码感官刺激、状态变化事件和动物自己的行为。一个显式的状态表示将对所有这些变量的结合进行编码，从而产生组合爆炸。建议的替代方法是让模型形成选定变量的连词，允许它在只关注与正在学习的任务相关的那些维度时，增量地扩展其状态描述。将开发基于快速单层神经网络学习的启发式方法，以根据最近的经验选择有用的连词。PI还将研究将工作记忆的当前状态与整个过去状态或事件的记录相匹配，以预测奖励。将开发一种灵活的架构，用于以参数化的方式表示动作（以便提供无限的可变性），并具有时间持续时间（允许在执行过程中到达刺激和奖励）。最后，将存在一些机制来应对行动的失败，以成功执行或产生预期奖励；这将为建模现象提供基础，如部分强化时间表的影响和灭绝期间增加的行为可变性。如果成功，这项工作将通过引入处理复杂状态和动作空间的新技术来推进强化学习的发展。这对动物认知理论、通过探索和实验学习的机器人以及向人类老师学习的机器人都具有重要意义。

The PI will develop a cognitively plausible reinforcement learning (RL) architecture as a model of instrumental learning in animals and robots. Although RL was initially inspired by animal learning phenomena, the field has since developed mainly by addressing AI concerns. A major limitation of current RL architectures as cognitive theories is the representation of state space. Models that maintain explicit state representations (such as Q~tables) are limited to simple domains with only a few variables, while models that represent the state space implicitly (e.g., using a neural net function approximator) require large amounts of training data and unreasonably long training times compared to real animals. The PI's approach is to develop specialized representations of state space that are appropriate for modeling animal behavior and can support desired generalizations. The simulated animal's working memory will encode sensory stimuli, state change events, and the animal's own actions. An explicit state representation would encode the conjunction of all these variables, generating a combinatorial explosion. The proposed alternative approach is for the model to form conjunctions of selected variables, allowing it to incrementally expand its state description while focusing on just those dimensions that are relevant to the task being learned. Heuristics based on fast, single-layer neural net learning will be developed to select useful conjunctions as a function of recent experience. The PI also will investigate matching the current state of working memory with records of entire past states, or episodes, in order to predict reward. A flexible architecture will be developed for representing actions in a parameterized manner (so as to provide infinite variability), and with temporal duration (allowing stimuli and rewards to arrive in the midst of execution). Finally, there will be mechanisms for coping with failure of an action to execute successfully or to produce an expected reward; this will provide the basis for modeling phenomena such as effects of partial reinforcement schedules and increased behavioral variability during extinction. If successful, this work will advance the state of the art of reinforcement learning by introducing new techniques for handling complex state and action spaces. This has important implications for theories of animal cognition, for robots that learn by exploration and experimentation, and for robots intended to learn from human teachers.