Scalable Bayesian Reinforcement Learning in the Games Industry

强化学习的主要挑战之一是确定良好的数据采样策略，有效地平衡探索所有可能策略的空间，并利用迄今为止产生更好结果的轨迹。在具有复杂状态和动作空间的环境中，例如在电子游戏中，这种挑战变得更加明显，因为传统的强化学习算法经常受到样本效率低下，模型偏差和过度拟合的影响。通过将不确定性估计和先验知识结合到学习过程中，贝叶斯强化学习自然地平衡了这种探索-利用的权衡，使其成为这些环境中应用的自然候选。然而，贝叶斯强化学习算法的计算量较大，这阻碍了它们的广泛采用。本研究将探讨如何扩展贝叶斯强化学习以处理大规模问题，同时保持计算效率和准确性。如果成功完成，本研究将导致在游戏行业中更有效和稳定的强化学习代理训练。通过这种方式，强化学习算法可以更容易地集成到设计管道中，从而实现更快，更稳定的开发，并增强用户体验。

One of the main challenges in reinforcement learning is identifying good data sampling strategies that effectively balance between exploring the space of all possible policies, and exploiting the trajectories that have yielded better outcomes so far. In environments with complex state and action spaces, such as those in video-games, this challenge becomes more apparent with traditional reinforcement learning algorithms often suffering from sample inefficiency, model bias, and over-fitting. Through incorporating uncertainty estimation and prior knowledge into the learning process, Bayesian reinforcement learning naturally balances this exploration-exploitation trade-off, making it a natural candidate for application in these environments. However, Bayesian reinforcement learning algorithms are more computationally intensive, which has hindered their wide-spread adoption. The proposed study will investigate methods to scale Bayesian reinforcement learning to handle large-scale problems, while maintaining computational efficiency and accuracy. On successful completion, this study will result in more efficient and stable training of reinforcement learning agents in the games industry. Through this, reinforcement learning algorithms can be more easily integrated into the design pipelines, resulting in quicker and more stable development as well as enhanced user experience.