Robust and Sample Efficient Reinforcement Learning

强化学习(RL)可以说是最全面的机器学习形式之一。它促进了主动学习，并允许系统在做出一系列决策时，在较长的一段时间内了解其环境。该系统还可以从可能被延迟的微弱信号中学习。这在机器人、自动驾驶汽车、对话代理、玩游戏、运筹学、自动交易、非近视推荐系统和自我管理网络中尤其有用。强化学习的一般性也使它变得复杂，因此在算法上具有挑战性。 目标：本工作的目标是开发算法，以提高强化学习的稳健性和样本效率。近年来，深度强化学习技术取得了巨大的进步，它可以扩展到高维输入(例如图像和自然语言)和复杂任务。然而，大多数成功仅限于具有模拟环境(例如，游戏、模拟机器人环境)的应用，因为当前算法可能执行昂贵的/灾难性的动作，并且可能需要与真实环境交互的大得令人望而却步的数据量。 方法：我将开发新的贝叶斯强化学习技术，可以量化环境的不确定性。这将有助于稳健性和样本效率。在贝叶斯学习中，在每个时间步估计和改进未知数的分布。这种分布还允许系统通过将其操作集中在环境中仍未知的部分来更有效地进行探索。为此，我将开发可扩展的贝叶斯技术，用于安全有效地进行深度强化学习。我还将开发新的受限强化学习技术，该技术考虑到次要目标，如不应超过的方差和成本函数。这将通过确保在工业应用中满足关键性能指标(KPI)来进一步提高稳健性。我还将开发对缺失输入具有健壮性的生成性强化学习技术。在一些应用中(例如，非近视推荐系统和自我管理网络)，某些观测/传感器可能在每个时间步长都不可用。将设计能够以原则性方式边缘化输入的生成性强化学习技术。

Reinforcement Learning (RL) is arguably one of the most comprehensive forms of machine learning. It facilitates active learning and it allows a system to learn over an extended period of time about its environment as it makes a sequence of decisions. The system can also learn from weak signals that might be delayed. This is particularly useful in robotics, autonomous vehicles, conversational agents, game playing, operations research, automated trading, non-myopic recommender systems and self-managing networks. The generality of reinforcement learning also makes it complex and therefore challenging algorithmically. Objectives: The goal of this work is to develop algorithms to improve the robustness and sample efficiency of reinforcement learning. Tremendous progress has been achieved in recent years by deep reinforcement learning techniques that scale to high dimensional inputs (e.g., images and natural language) and complex tasks. However, most of the successes are limited to applications with simulated environments (e.g., games, simulated robotic environments) since current algorithms may execute costly/catastrophic actions and may require an amount of data that is prohibitively large for interaction with real environments. Methods: I will develop novel Bayesian reinforcement learning techniques that can quantify the uncertainty of the environment. This will be helpful both for robustness and sample efficiency. In Bayesian learning, a distribution over the unknowns is estimated and refined at each time step. This distribution also allows a system to explore more efficiently by focusing its actions on the parts of the environment that are still unknown. To that effect, I will develop scalable Bayesian techniques for deep reinforcement learning that explore safely and efficiently. I will also develop novel constrained reinforcement learning techniques that take into account secondary objectives such as variance and cost functions that should not be exceeded. This will further improve robustness by ensuring that key performance indicators (KPIs) are met in industrial applications. I will also develop generative reinforcement learning techniques that are robust to missing inputs. In some applications (e.g., non-myopic recommender systems and self-managing networks), some observations/sensors might not be available at each time step. Generative reinforcement learning techniques that can marginalize inputs in a principled way will be designed.