Reinforcement Learning for Finite Horizons (ReLeaF)

强化学习（RL）是一种学习如何在最初未知的环境中采取行动以优化预期结果的技术，它通过最大化累积奖励的概念来建模。将目标写成时间规范的学习算法有三个关键要素：从规范到适当的有限自动机的转换；将这些有限自动机转化为奖励结构，使提供最优奖励的策略保证提供最优控制；并将其包装成一个折扣方案，对于适当的参数，将确保学习者收敛到最优策略。我们将考虑一种用于自动化和运动规划的流行规范语言的RL问题，即有限视界线性时间时间逻辑ltf。特别是，我们将研究无模型强化学习算法，它比基于模型的算法更适合于难以预测环境行为的现实应用。我们将提出学习算法，提供从有限视界LTL到奖励结构的转换，并对建模为马尔可夫决策过程（mdp）的环境提供满足给定目标的正式保证。我们将把我们的技术扩展到无限状态的mdp，包括可以提供形式保证的变化——比如可数的、有限分支的mdp——并研究我们的技术在更一般的类中提供保证的条件，比如紧凑mdp的平滑保证。我们将通过研究有约束的目标来补充这些研究方向。这是有效地考虑优先目标，其中满足安全约束是优先考虑的，而其他属性（如效率）则被认为是提供相同安全保证的策略之间的决定性因素。

Reinforcement learning (RL) is a technique for learning how to take actions in an initially unknown environment in order to optimise an expected outcome, which is modelled through the notion of maximising an accumulative reward. Learning algorithms with goals written as temporal specifications have three key ingredients: the translation from the specification to appropriate finite automata; the translation of these finite automata to reward structures, such that a strategy that provides optimal rewards is guaranteed to provide optimal control; and a wrapper into a discounting scheme that, for appropriate parameters, will ensure that a learner converge to an optimal strategy.We will consider the RL problems for a popular specification language used in automation and motion planning, the finite horizon linear time temporal logic LTLf. In particular, we will study model-free RL algorithms, which are more suitable to real-world applications where the behaviour of the environment is hard to predict, than its model-based counterpart. We will propose learning algorithms that provide translations from finite horizon LTL to reward structures with formal guarantees of satisfying the given goals for environments modelled as Markov Decision Processes (MDPs). We will extend our techniques to infinite-state MDPs, including variations where formal guarantees can be provided -- like countable, finitely branching MDPs -- and study conditions for our techniques to provide guarantees in more general classes, such as smoothness guarantees for compact MDPs. We will complement these lines of research by looking at goals with constraints. This is effectively considering prioritised goals, where meeting safety constraints takes precedence, while other properties -- such as efficiency -- are considered as tie-breakers among strategies that provide the same safety guarantees.