RI: Small: Non-parametric Approximate Dynamic Programming for Continuous Domains

这个项目涉及一种被称为强化学习的机器学习技术，它与心理学中使用的强化学习的概念有关，但又不同。共同的元素是，这两种观点都研究了由经验导致的行为变化。在机器学习的情况下，行为通常是在动态环境中做出决策，例如控制机器人、工厂、仓库的库存水平，甚至药物剂量水平。目前这一领域的理论发展保证了强化学习算法可以做出最优决策，但只有在实践中难以确保的限制性假设下才能做出最优决策。将强化学习应用于重大实际问题的努力已经取得了一些成功，但这种努力往往放弃了理论上的保证，而依赖于专家乏味的参数调整(人类试验和错误)来取得成功。本研究试图减少使强化学习成功所需的人类试验和错误的量，从而使其成为更广泛的人更容易获得的工具。具体地说，它将专注于由连续变量描述的领域的算法，寻求为这些领域提供更强有力的理论保证，以及一种平衡尝试新事物的预期好处与坚持已知问题(探索与利用)的好处的方法。这一领域成功的一个实际好处是改进的技术，使人们更容易在各种实际任务中部署学习算法并提高性能，如上面提到的那些：机器人或工厂控制、库存管理或药品运送。该项目计划使用模型直升机作为挑战领域，但它与直升机控制本身无关。相反，它寻求开发可以应用于许多问题的通用技术，包括直升机，并将使用模型直升机作为一种廉价而有趣的方式来激励学生。该项目旨在开发一个模型直升机模拟器(以降低在实际直升机上尝试一切的成本和风险)，并计划将该模拟器提供给研究界，提供一个有趣和具有挑战性的基准问题。

This project concerns a machine learning technique known as reinforcement learning, which is related to, but distinct from, the notion of reinforcement learning used in psychology. The common element is that both views study changes in behavior that result from experience. In the machine learning case, the behaviors are often decision making in dynamic environments, such as controlling a robot, a factory, inventory levels for a warehouse or even drug dosage levels. Current theoretical development in this area guarantees that optimal decisions can be made by reinforcement learning algorithms, but only under restrictive assumptions that are difficult to ensure in practice. Efforts to apply reinforcement learning to significant practical problems have enjoyed some success, but such efforts often forgo theoretical guarantees and rely upon tedious parameter adjustments by experts (human trial and error) to achieve success.This research seeks to reduce the amount of human trial and error needed to make reinforcement learning successful, thereby making it a more accessible tool to a wider range of people. Specifically, it will focus on algorithms for domains described by continuous variables, seeking to provide stronger theoretical guarantees for such domains as well as an approach that balances the anticipated benefit of trying new things with the benefit of sticking to what is already known about a problem (exploration vs. exploitation). A practical benefit of success in this area would be improved techniques that make it easier for people to deploy algorithms that learn and improve performance in a variety of practical tasks like those mentioned above: robot or factory control, inventory management, or drug delivery.This project plans to use a model helicopter as a challenge domain, but it is not about helicopter control per se. Rather, it seeks to develop general techniques that can apply to many problems, including helicopters, and will use model helicopters as an inexpensive and fun way to motivate students. The project aims to develop a model helicopter simulator (to reduce the cost and risk of trying everything on an actual helicopter) and plans to make this simulator available to the research community, providing a fun and challenging benchmark problem.