CIF: Small: Compression Schemes for Communication Constrained Bandit and Reinforcement Learning

CIF：小：通信受限强盗和强化学习的压缩方案

• 批准号: 2221871
• 负责人: Lin Yang
• 金额: $ 60万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2221871&HistoricalAwards=false
• 关键词: CIF; Small; Compression; Schemes; Communication

Active learning and online learning are machine-learning paradigms in which computers learn to make complex decisions while receiving feedback from an environment. For instance, a drone may learn to fly by itself, or a car may learn to drive by trial and error. Recently, these learning paradigms have been widely applied and have achieved phenomenal successes with human-level performance in tasks like gameplay or robot control. As computing devices become smaller and less power-consuming, new distributed learning frameworks start to emerge. These frameworks contain low-capability learning agents (such as cell phones, unmanned vehicles, or drones) that are far apart but perform learning collectively by communicating with each other through (wireless) networks. However, existing communication approaches would become bottlenecks for learning since they were designed for high-power computers and consume too much power and network bandwidth. This project aims to address this issue by providing novel techniques that efficiently compress data to be communicated while preserving the learning ability. The techniques developed in this project will advance the state-of-the-art in distributed online/active learning by improving communication efficiencies. The overarching goal of this project is to establish efficient compression schemes that support effective active/online learning, such as bandit and reinforcement learning over communication-constrained networks. In these learning environments, a learner aims to make a good decision for the next steps based on experience; this project will explore fundamental bounds and efficient algorithms that support this goal while minimizing the number of bits communicated - by compressing in a way that only retains the necessary information for decision making. In other words, this project aims to explore the fundamental trade-off between compression and learnability in active/online environments. Building on promising preliminary work, the investigators will study problems ranging from the most basic multi-arm bandit setting to more complex reinforcement learning settings and consider both centralized and decentralized network topologies. More specifically, the investigators propose compression schemes and fundamental theoretical bounds for (1) rewards in multi-armed bandit problems, (2) context vectors for contextual bandit problems, and (3) state-action features and models for Markov decision problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

主动学习和在线学习是机器学习的范例，其中计算机在接收来自环境的反馈的同时学习做出复杂的决策。例如，无人机可以自己学习飞行，或者汽车可以通过试验和错误来学习驾驶。最近，这些学习范式已被广泛应用，并在游戏或机器人控制等任务中取得了惊人的成功。随着计算设备变得越来越小，功耗越来越低，新的分布式学习框架开始出现。这些框架包含低能力的学习代理（如手机，无人驾驶汽车或无人机），这些代理相距甚远，但通过（无线）网络相互通信来共同执行学习。然而，现有的通信方法将成为学习的瓶颈，因为它们是为高功率计算机设计的，消耗太多的功率和网络带宽。该项目旨在通过提供新的技术来解决这个问题，这些技术可以有效地压缩要通信的数据，同时保留学习能力。在这个项目中开发的技术将通过提高通信效率来推进分布式在线/主动学习的最新技术。该项目的总体目标是建立有效的压缩方案，支持有效的主动/在线学习，如通信受限网络上的强盗和强化学习。在这些学习环境中，学习者的目标是根据经验为下一步做出好的决定;本项目将探索支持这一目标的基本界限和有效算法，同时最大限度地减少通信的比特数-通过压缩的方式，只保留决策所需的信息。换句话说，该项目旨在探索主动/在线环境中压缩和可学习性之间的基本权衡。在有希望的初步工作的基础上，研究人员将研究从最基本的多臂强盗设置到更复杂的强化学习设置的问题，并考虑集中式和分散式网络拓扑。更具体地说，研究人员提出的压缩方案和基本理论界限（1）奖励多臂强盗问题，（2）上下文的强盗问题的上下文向量，（3）状态-动作功能和马尔可夫决策问题的模型。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Near-Optimal Sample Complexity Bounds for Constrained MDPs

受限 MDP 的近乎最优样本复杂度界限

• 发表时间: 2022
• 期刊: Advances in neural information processing systems
• 作者: Vaswani, Sharan;Yang, Lin;Szepesvári, Csaba
• 通讯作者: Szepesvári, Csaba

Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning

• DOI: 10.48550/arxiv.2304.08944
• 发表时间: 2023-04
• 期刊: ArXiv
• 作者: Dingwen Kong;Lin F. Yang

PROVABLY EFFICIENT LIFELONG REINFORCEMENT LEARNING WITH LINEAR REPRESENTATION

具有线性表示的可证明有效的终身强化学习

• 发表时间: 2023
• 期刊: ICLR
• 作者: Amani, Sanae;Yang, Lin;Cheng, Ching-An
• 通讯作者: Cheng, Ching-An

Low-Switching Policy Gradient with Exploration via Online Sensitivity Sampling

• DOI: 10.48550/arxiv.2306.09554
• 发表时间: 2023-06
• 期刊: ArXiv
• 作者: Yunfan Li-;Yiran Wang-;Y. Cheng;Lin F. Yang

Horizon-Free Learning for Markov Decision Processes and Games: Stochastically Bounded Rewards and Improved Bounds

马尔可夫决策过程和博弈的无地平线学习：随机有界奖励和改进界限

• 发表时间: 2023
• 期刊: Proceedings of Machine Learning Research
• 作者: Li, Shengshi;Yang, Lin
• 通讯作者: Yang, Lin