Task-general reinforcement learning algorithms

该项目属于EPSRC信息和通信技术（ICT）研究领域。该项目的目标是开发能够从数据中提取任务一般结构的算法，并使用该结构有效地学习新任务。这些算法将能够在现实世界中部署强化学习代理，在那里他们可以根据很少的经验获得新技能。这种数据高效的技能获取可能会导致工业过程和家庭日常更经济的自动化。虽然强化学习中的任务泛化并不是一个新的研究课题，但有效地解决它需要从最近在该领域流行的单任务视角缩小范围。从这个角度出发的第一步是考虑将新任务的泛化性能作为成功的关键衡量标准。这被作为本项目中开发的算法的中心评价标准。评估将包括收集对算法的经验和理论支持。开发和评估的结果发表在人工智能和机器学习社区的顶级会议和学术期刊上。提高强化学习中任务泛化的一种方法是考虑一个更高层次的学习问题，称为元学习，其目的是学习学习算法本身，明确目标是在新任务上快速学习。元学习是一种很有前途的任务泛化工具，因为它可以通过将泛化问题转化为学习，在数据丰富的情况下利用深度学习的优势。虽然元学习在过去几年中引起了人们的兴趣，并做出了许多令人兴奋的贡献，但这些方法在训练任务分布之外的真正新颖任务的泛化性能只获得了有限的关注。这种缺乏关注作为一个路标，引导这个研究项目进入相对未被探索的领域，即明确地解决任务泛化问题。具体来说，在这个项目中，为任务-通用强化学习开发了新的算法和训练环境。为了开发新的算法，将考虑新的元参数化强化学习代理和算法本身。强化学习智能体的泛化性能不仅依赖于训练算法，还依赖于训练环境和数据集。因此，必须发展新的以泛化为重点的训练环境。

This project falls within the EPSRC Information and communication technologies (ICT) research area. The goal of the project is to develop algorithms capable of extracting task-general structure from data and using that structure to efficiently learn on novel tasks. These algorithms would enable deployment of reinforcement learning agents in the real world, where they could acquire new skills based on little experience. This data efficient skill acquisition could lead to more economical automation of industrial processes and household routines. While task-generalization in reinforcement learning is not a new research topic, effectively tackling it requires zooming out from the single-task perspective that has been prevalent in the field recently. The first steps away from that perspective are taken by considering generalization performance to novel tasks a key measure of success. This is adopted as a central evaluation criterion for the algorithms developed in this project. The evaluation will consist of gathering both empirical and theoretical support for the algorithms. The outcomes of the development and evaluation are published in the top conferences and academic journals of the artificial intelligence and machine learning communities.One way to improve task-generalization in reinforcement learning is to consider a higher-level learning problem, called meta-learning, which aims to learn the learning algorithm itself with the explicit objective of fast learning on novel tasks. Meta-learning is a promising tool for task-generalization since it enables leveraging the strength of deep learning when abundant data is available by turning the problem of generalization also into learning. While meta-learning has seen a surge of interest and many exciting contributions in the past few years, the generalization performance of these approaches to genuinely novel tasks outside the training task distributions has garnered only limited attention. This lack of attention serves as a signpost guiding this research project into the relatively unexplored territory of tackling the questions of task-generalization explicitly.Concretely in this project, new algorithms and training environments are developed for task-general reinforcement learning. To develop new algorithms, novel meta-parameterizations of reinforcement learning agents and the algorithms themselves will be considered. The generalization performance of reinforcement learning agents does not only rely on the training algorithm but on the training environments and datasets as well. Therefore, new generalization-focused training environments will have to be developed.