CRII: RI: Accelerated Stochastic Approximation for Reinforcement Learning

该项目开发了一类新的强化学习加速学习技术。强化学习是一种通过与未知环境进行试错交互来实现自主决策的方法，重点是从数据流中进行增量学习。强化学习具有巨大的工业潜力，特别是对于实时控制系统，例如能源和搜索和救援机器人的主动网络管理，并且已经在广泛的领域中使用，包括机器人，心理学，动物学习和神经科学。为了提高强化学习的实际应用，该项目提出了一类新的算法，其目标是平衡计算复杂度和学习的样本效率，这通常需要大量的计算和内存。这种试图平衡这两种要求的算法空间在强化学习中尚未得到充分探索，并为影响工业应用和不断增长的计算可持续性领域提供了令人兴奋的机会。该项目的一个重要方面是在广泛的模拟环境中实施和研究这些算法，并通过课程和夏季研究吸引不同群体的学生。该项目开发了有效的增量近似来总结收集的样本，以提高样本效率，并开发了一个经验框架来评估这些算法。这类新的加速学习技术形式上权衡了计算和精度，并通过各种加速随机梯度下降技术和增量矩阵近似，具有许多有前途的扩展和研究方向。此外，另一个重点是为强化学习社区开发工具和新措施，以评估样本效率和计算复杂性之间的平衡，并通过现有的开源平台发布代码框架。对这些新的优化变体的初步系统探索将为提高工业中强化学习的有效性和实用自主代理的长期目标奠定基础。

This project develops a new class of accelerated learning techniques for reinforcement learning. Reinforcement learning is an approach to autonomous decision-making through trial-and-error interaction with an unknown environment, with a focus on learning incrementally from this stream of data. Reinforcement learning has significant industrial potential, particularly for real-time control systems, such as active network management for energy and search-and-rescue robots, and is already used in a wide range of fields, including robotics, psychology, animal learning and neuroscience. To improve the practical application of reinforcement learning, this project proposes a new class of algorithms with the goal to balance computational complexity and the sample efficiency of learning, which often requires significant computation and memory. This space of algorithms that attempt to balance both requirements has been under-explored for reinforcement learning, and provide exciting opportunities to impact industrial applications and the growing area of computational sustainability. An important aspect of this project will be to implement and study these algorithms on a wide-range of simulated environments, and engage a diverse group of students through courses and summer research.This project develops efficient incremental approximations to summarize gathered samples for improved sample efficiency and an empirical framework to evaluate these algorithms. This new class of accelerated learning techniques formally trade-off computation and accuracy and have many promising extensions and research directions, through a variety of accelerated stochastic gradient descent techniques and incremental matrix approximations. Further, another focus is to develop tools and novel measures for the reinforcement learning community that evaluate this balance between sample efficiency and computational complexity, with the code framework released through an existing open-source platform. This initial systematic exploration of these novel optimization variants will lay the foundation for the long-term goal of improving efficacy of reinforcement learning in industry and for practical autonomous agents.