Developing robust and scalable reinforcement learning algorithms

点击翻译按钮获取中文摘要

Reinforcement learning (RL) represents a powerful paradigm for applying machine learning to complex decision making tasks. However, current RL algorithms are brittle and have numerous failure mechanisms, making them difficult to apply to beyond the synthetic simple environments used in research. Over my PhD I aim to help in developing robust and scalable RL algorithms, such that RL can be more easily applied to large, complex, real-world problems.One promising approach to this is offline RL; a broad set of methods which utilize pre-existing datasets to remove the need for the online data collection required by standard RL methods. Use of large-scale datasets has been the driver of the transformative recent advances in supervised learning, and offline RL presents a way of enabling similar scaling progress in RL. In addition to this, offline RL is a practical choice in many real-world domains where online data collection is costly or poses safety risks, such as in robotics or healthcare. The offline approach, however, introduces the extra problem of needing to evaluate effectiveness of actions or behaviour not covered in the dataset. This additional challenge has been shown to cause existing online RL algorithms to fail, and hence represents a significant barrier to robust, scalable offline RL algorithms.In my recent project we investigated this issue and identified the "edge-of-reach" failure mechanism in offline model-based reinforcement learning. Based on these insights we were able propose a "value uncertainty-aware" approach which resolves this issue. As an extension to this, I plan to investigate whether meta-learning can be used to discover an offline RL algorithm that deals with value uncertainty even more effectively.Another barrier to large-scale RL is the phenomenon of "plasticity loss," whereby the inherent non-stationarity in RL training has been shown to cause deep RL agents to gradually lose the ability to learn from new data. I am currently investigating the underlying mechanisms behind this phenomenon, with the hope that this will inform development of more effective mitigation techniques.Finally, the recent advances in large language models present huge opportunity for using language as a tool for instilling real-world knowledge and inductive biases into RL agents. The most effective approach for this integration is an open question, and I plan to work on exploring this. In summary, RL has huge potential which is not yet realized by current approaches, and during my PhD I aim to contribute towards developing RL algorithms which are robust and scalable and are effective when applied to complex real-world problems.