A Comparative Tutorial of Bayesian Sequential Design and Reinforcement Learning
A Comparative Tutorial of Bayesian Sequential Design and Reinforcement Learning
复制标题
DOI：
10.1080/00031305.2022.2129787
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Mauricio Tec;Yunshan Duan;P. Müller
中科院分区：
文献类型：
--
作者：
Mauricio Tec;Yunshan Duan;P. Müller
摘要强化学习(RL)是序列决策问题中报酬驱动学习的一种计算方法。它通过向与环境交互的代理学习而不是从受监督的数据学习来实现最优操作的发现。将RL与传统的序贯设计进行了对比和比较，重点研究了基于仿真的贝叶斯序贯设计。最近，人们对RL技术在医疗保健应用中的兴趣与日俱增。作为激励的例子，我们介绍了两个相关的应用。在这两种应用中，判决的顺序性质仅限于顺序停止。讨论的焦点不是全面的调查，而是使用标准工具解决这两个相对简单的顺序停车问题的解决方案。这两个问题都受到适应性临床试验设计的启发。我们使用示例来解释构成每个框架的术语和数学背景，并将它们映射到另一个框架。实现和结果说明了RL和BSD之间的许多相似之处。这些结果激发了对每种方法的潜在优势和局限性的讨论。
Abstract Reinforcement learning (RL) is a computational approach to reward-driven learning in sequential decision problems. It implements the discovery of optimal actions by learning from an agent interacting with an environment rather than from supervised data. We contrast and compare RL with traditional sequential design, focusing on simulation-based Bayesian sequential design (BSD). Recently, there has been an increasing interest in RL techniques for healthcare applications. We introduce two related applications as motivating examples. In both applications, the sequential nature of the decisions is restricted to sequential stopping. Rather than a comprehensive survey, the focus of the discussion is on solutions using standard tools for these two relatively simple sequential stopping problems. Both problems are inspired by adaptive clinical trial design. We use examples to explain the terminology and mathematical background that underlie each framework and map one to the other. The implementations and results illustrate the many similarities between RL and BSD. The results motivate the discussion of the potential strengths and limitations of each approach.