Learning in embedded systems
Learning in embedded systems
复制标题
DOI：
10.7551/mitpress/4168.001.0001
复制
发表时间：
1993-05
期刊：
影响因子：
--
通讯作者：
L. Kaelbling
中科院分区：
文献类型：
--
作者：
L. Kaelbling
本论文解决了嵌入式系统中学习算法的设计问题。这个问题与传统的监督学习问题不同。代理发现自己处于特定的输入情况时必须生成一个动作。然后，它从环境中接收强化值，表明环境的当前状态对于代理来说有多有价值。然而，代理无法推断出执行任何其他操作所产生的强化值。提出了许多根据强化值学习行动策略的算法，并与现有的强化学习算法进行了实证比较。区间估计算法使用置信区间的统计概念来指导其在世界中的行动的生成，权衡获取信息的行动与获得强化的行动。它在简单领域表现良好，但没有表现出任何泛化性，并且计算复杂。级联算法是一种结构性信用分配方法，允许通过学习布尔函数的强化学习模块集合来学习具有许多输出位的动作策略。该方法代表了计算复杂性和学习率的改进。描述了在 k-DNF 中学习布尔函数的两种算法。两者都基于 Valiant 的算法，用于从输入输出实例中学习此类函数。第一个使用萨顿的线性关联和强化比较技术，而第二个使用区间估计算法的技术。它们都表现良好并且具有易于处理的复杂性。提出了一种生成和测试强化学习算法。它允许增量构建布尔函数的符号表示并在环境中进行测试。它是高度参数化的，可以调整以学习广泛的函数类。即使存在大量不相关的输入位，也可以非常有效地学习低复杂度函数。该算法扩展为使用设置重置运算符构建简单的顺序网络，这允许代理学习具有状态的动作策略。这些算法除了在仿真中进行研究外，还在物理移动机器人上进行了实现和测试。
This dissertation addresses the problem of designing algorithms for learning in embedded systems. This problem differs from the traditional supervised learning problem. An agent, finding itself in a particular input situation must generate an action. It then receives a reinforcement value from the environment, indicating how valuable the current state of the environment is for the agent. The agent cannot, however, deduce the reinforcement value that would have resulted from executing any of its other actions. A number of algorithms for learning action strategies from reinforcement values are presented and compared empirically with existing reinforcement-learning algorithms. The interval-estimation algorithm uses the statistical notion of confidence intervals to guide its generation of actions in the world, trading off acting to gain information against acting to gain reinforcement. It performs well in simple domains but does not exhibit any generalization and is computationally complex. The cascade algorithm is a structural credit-assignment method that allows an action strategy with many output bits to be learned by a collection of reinforcement-learning modules that learn Boolean functions. This method represents an improvement in computational complexity and often in learning rate. Two algorithms for learning Boolean functions in k-DNF are described. Both are based on Valiant's algorithm for learning such functions from input-output instances. The first uses Sutton's techniques for linear association and reinforcement comparison, while the second uses techniques from the interval estimation algorithm. They both perform well and have tractable complexity. A generate-and-test reinforcement-learning algorithm is presented. It allows symbolic representations of Boolean functions to be constructed incrementally and tested in the environment. It is highly parametrized and can be tuned to learn a broad range of function classes. Low-complexity functions can be learned very efficiently even in the presence of large numbers of irrelevant input bits. This algorithm is extended to construct simple sequential networks using a set-reset operator, which allows the agent to learn action strategies with state. These algorithms, in addition to being studied in simulation, were implemented and tested on a physical mobile robot.