RTML: Small: Real-Time Model-Based Bayesian Reinforcement Learning

许多机器学习技术的适用性仅限于非常具体、固定的应用环境。与此相反，基于模型的贝叶斯强化学习技术是一个更复杂的框架，可以在不完全了解环境的情况下，通过不断变化的目标实现类似人类的决策。然而，它的计算复杂度非常高，因此，它的进展仅限于有限应用的小规模演示。该项目的目标是通过开发创新的算法-硬件协同设计技术来克服这一障碍。该研究成果有望大大加快基于模型的贝叶斯强化学习的计算速度，用于实际的、大规模的现实应用，特别是那些具有实时约束的应用。研究成果将惠及无人驾驶汽车、无人机、智能农业灌溉、救灾机器人、残疾人机器人助理等直接影响社会的领域。这项研究还将为相关技术领域急需的美国劳动力培训学生，包括女性和其他代表性不足的群体。基于模型的贝叶斯强化学习的计算核心是决策树上的随机抽样。在这个项目中，计算加速将利用采样过程提供的内在并行性来实现。将解决传统方法的内存和算术瓶颈。首先，将开发一种基于逻辑电路的技术来表示概率，从而大大降低算法的内存利用率。其次，将探索强大的新算术技术，以实现新提出的数字表示的面积高效计算。第三，研究一种便于电路实现的新采样方法。第四，将研究一种近似技术，以减轻跟踪采样历史所带来的复杂性。最后，利用路径感知并行采样来避免软件实现中的冗余计算。这些技术及其整体有效性将通过实验进行验证。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The applicability of many machine-learning techniques is limited to very specific, fixed application environments. As opposed to this, model-based Bayesian reinforcement learning technique is a more sophisticated framework, enabling human-like decision-making with evolving objectives and without complete knowledge of the environment. However, its computational complexity is very high, and consequently, its progress has been limited to small-scale demonstrations on limited applications. The goal of this project is to overcome this hurdle by developing innovative algorithm-hardware co-design techniques. The research outcome is expected to greatly accelerate the computation of model-based Bayesian reinforcement learning for practical, large-scale real-life applications, especially those with real-time constraints. The research results will benefit many fields that directly impact society, e.g., driver-less cars, unmanned aerial vehicles, smart agricultural irrigation, robotics for disaster relief and robotic assistants for handicapped people. The research will also train students, including women and other underrepresented groups, for the much needed U.S. workforce in related areas of technology.The computational kernel of model-based Bayesian reinforcement learning is random sampling over a decision tree. In this project, the computational acceleration will be realized by making use of the intrinsic parallelism offered by the sampling process. Both the memory and the arithmetic bottlenecks of traditional approaches will be addressed. First, a logic circuit based technique will be developed to represent probabilities, and thereby greatly reduce the memory utilization of the algorithm. Second, powerful new arithmetic techniques will be explored to achieve area-efficient computations for the newly proposed number representation. Third, a new sampling method, that is friendly to circuit implementation, will be investigated. Fourth, an approximation technique will be studied to alleviate the complexity arising from tracing sampling histories. Finally, path-aware parallel sampling will be exploited to avoid the redundant computations in software implementations. These techniques and their overall effectiveness will be validated via experiments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.