A Sample-Efficient Black-Box Optimizer to Train Policies for Human-in-the-Loop Systems With User Preferences
A Sample-Efficient Black-Box Optimizer to Train Policies for Human-in-the-Loop Systems With User Preferences
复制标题
一种样本高效的黑盒优化器，用于根据用户偏好来训练人在环系统的策略
DOI：
--
复制
发表时间：
2017
影响因子：
5.2
通讯作者：
H. Geyer
中科院分区：
文献类型：
--
作者：
Nitish Thatte;Helei Duan;H. Geyer
提出了一种基于定性偏好反馈的人在环系统控制策略优化算法。该方法特别适用于下肢假体和外骨骼等难以定义目标函数、难以识别模型、重复硬件实验成本高的系统。为了解决这些问题，我们结合并扩展了一种基于偏好的学习算法和预测熵搜索贝叶斯优化方法。由此产生的算法，带有偏好的预测熵搜索（PES-P），在控制参数集对之间征求偏好，以最少的实验次数最优地减少目标函数最优分布的不确定性。在从优化随机生成的函数到调整线性系统和行走模型的控制参数的三个模拟测试中，我们发现该算法优于预期的改进方法（EI）和通过拉丁超立方体（LH）进行的随机比较。此外，我们在一项关于机器人经股假体控制的初步研究中发现，在给定真实用户偏好的情况下，PES-P比EI或LH更快、更一致地找到良好的控制参数。结果表明，所提出的算法可以帮助工程师更准确、有效和一致地优化某些机器人系统。
We present a new algorithm for optimizing control policies for human-in-the-loop systems based on qualitative preference feedback. This method is especially applicable to systems such as lower limb prostheses and exoskeletons for which it is difficult to define an objective function, hard to identify a model, and costly to repeat hardware experiments. To solve these problems, we combine and extend an algorithm for learning from preferences and the Predictive Entropy Search Bayesian optimization method. The resulting algorithm, Predictive Entropy Search with Preferences (PES-P), solicits preferences between pairs of control parameter sets that optimally reduce the uncertainty in the distribution of objective function optima with the least number of experiments. We find that this algorithm outperforms the expected improvement method (EI), and random comparisons via Latin hypercubes (LH) in three simulation tests that range from optimizing randomly generated functions to tuning control parameters of linear systems and of a walking model. Furthermore, we find in a pilot study on the control of a robotic transfemoral prosthesis that PES-P finds good control parameters quickly and more consistently than EI or LH given real user preferences. The results suggest the proposed algorithm can help engineers optimize certain robotic systems more accurately, efficiently, and consistently.