research into Reformulating reinforcement learning

强化学习(RL)可以被视为在未知环境中的优化，其目标是平衡探索(访问环境中的新状态/动作)和开发(重新访问状态和动作，并获得巨大回报)。RL的力量在最近几年已经被多次证明，例如AlphaGo击败了世界上最好的围棋选手。本项目的目标是评估最近一种方法[1]对RL的适用性，以表示不确定性的双重性质：随机和确定性。前者指的是通常的概率方法，后者对应于某些东西(例如，统计模型中的参数)是固定但未知的情况。这与RL相关，在RL中，环境可能是真正随机的或只是未知的，例如对手的策略或给定动作的奖励概率。[1]的框架基于外部测量的测度论概念，自然地弥合了频率论和贝叶斯方法之间的差距。这对于目前两种方法共存的RL来说也是至关重要的。这里有两个主要的研究问题：1)在性能和计算效率方面引入更真实的不确定性表示有什么实际好处？2)为频率和贝叶斯技术发展的理论保证能否扩展到基于外部测量的算法？为了回答这些问题，首先将考虑多臂强盗(MAB)的特殊情况。单抗足够简单，可以得到理论上的保证，同时仍然提出了关于勘探和开采的RL的根本困境。一类外部概率度量的参数估计.arxiv：1801.00569,2018.海军集团是一家专门从事海军防御的法国公司，MABS和RL解决的顺序资源分配问题出现在军事行动的许多关键方面。例如，军用舰艇配备了一系列传感器，可以在不同的模式下工作；控制这些传感器来实现不同的目标是一个具有挑战性的问题，需要处理一个存在不同类型不确定性的复杂环境。研究-强化学习(RL)的背景可以被视为在未知环境中的优化，其中目标是平衡探索(访问环境中的新状态/动作)和开发(重新访问环境中的状态和动作)。研究的目的和目标-本项目的目标是评估基于可能性和概率理论的最新方法对RL的适用性。这与RL相关，在RL中，环境可能是真正随机的或简单地未知的，例如对手的策略或给定行动的奖励概率。研究方法的新颖性-提出的方法基于贝叶斯推理的新公式，使用可能性理论的工具来建模参数不确定性。潜在的影响、应用和好处--通过扩展标准统计框架，建议的方法有可能为MAB和一般的RL带来新的解决方案。在统计学中，所考虑的方法允许解释几个标准的启发式方法，从而提供正式的基础来理解它们的性质和限制。因此，有可能为现有技术带来新的见解。研究如何与职权范围相关--拟议的研究是在EPSRC的数学科学主题下进行的，特别是在统计和应用概率项下以及在理论计算机科学项下。研究领域：数学科学外部合作伙伴-海军小组

Reinforcement learning (RL) can be viewed as optimisation in an unknown environment, where the goal is to balance exploration (visiting new states/actions in the environment) and exploitation (revisiting states and actions with large rewards). The power of RL has been demonstrated many times in the recent years, for instance with AlphaGo defeating the world's best Go player.The objective of this project is to assess the applicability to RL of a recent approach [1] for representing the dual nature of uncertainty: random and deterministic. The former refers to the usual probabilistic approach and the latter corresponds to the case where something (e.g. a parameter in a statistical model) is fixed but unknown. This is relevant for RL where the environment can be truly random or simply unknown, such as with the strategy of an opponent or with the probability of reward of a given action.The framework of [1], based on the measure-theoretic notion of outer measure, naturally bridges the gap between the frequentist and Bayesian approaches. This can also be crucial for RL where the two approaches currently coexist.There are two main research questions:1)What are the practical benefits of introducing a more faithful representation of uncertainty in terms of performance and computational efficiency?2)Can the theoretical guarantees developed for frequentist and Bayesian techniques be extended to algorithms based on outer measures?To answer these questions, the special case of the multi-armed bandits (MABs) will first be considered. MABs are sufficiently simple to allow for theoretical guarantees to be derived while still presenting the fundamental dilemma of RL regarding exploration vs exploitation.[1] J. Houssineau. Parameter estimation with a class of outer probability measures.arXiv:1801.00569, 2018.Naval Group is a French company specialised in naval-based defence.The sequential resource allocation problems that MABs and RL solve appear in a number of crucial aspects in military operations. For instance, military vessels are equipped with a range of sensors that can operate in various modes; controlling these sensors to fulfil different objectives is a challenging problem that requires dealing with a complex environment where different types of uncertainty arise.The context of the research - Reinforcement learning (RL) can be viewed as optimisation in an unknown environment, where the goal is to balance exploration (visiting new states/actions in the environment) and exploitation (revisiting states and actions with large rewards).The aims and objectives of the research - The objective of this project is to assess the applicability to RL of a recent approach based on a combination of possibility and probability theory. This is relevant for RL where the environment can be truly random or simply unknown, such as with the strategy of an opponent or with the probability of reward of a given action.The novelty of the research methodology - The proposed approach is based on a new formulation of Bayesian inference using the tools of possibility theory to model parameter uncertainty. This approach lends itself to RL problems where the initial absence of knowledge must be faithfully represented.The potential impact, applications, and benefits - By extending the standard statistical framework, the proposed approach has the potential to lead to new solutions for MABs and for RL in general. Within statistics, the considered approach allows for explaining several standard heuristics and hence providing formal ground to understand their properties and limitations. There is therefore a potential for bringing new insights into existing techniques.How the research relates to the remit - The proposed research is under the Mathematical Sciences theme of the EPSRC, in particular under Statistics and Applied Probability and under Theoretical Computer Science.Research Area: Mathematical SciencesExternal Partner - Naval Group