Transfer of Samples in Policy Search via Multiple Importance Sampling
Transfer of Samples in Policy Search via Multiple Importance Sampling
复制标题
DOI：
--
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Andrea Tirinzoni;Mattia Salvini;Marcello Restelli
中科院分区：
文献类型：
--
作者：
Andrea Tirinzoni;Mattia Salvini;Marcello Restelli
算法1需要ESS的测量，以评估梯度估计的质量，从而适应批量大小。尽管已经研究了IS的几种ESS措施（参见，例如，（Martino等人，2017）），据我们所知，没有提出专门为MIS估计器设计的措施。Elvira et al.（2018）最近的一项工作分析了第2节中介绍的经典ESS测量，并根据经验证明了其在MIS中的有效性。因此，我们决定也将其应用于我们的上下文。然而，由于对于我们的应用，我们对ESS的下限感到满意，而不是在给定的建议下考虑重要性权重的方差，我们将其作为w.r.t.这些的混合物。这是由下面的命题所激发的，该命题直接从前一个方差总是小于后一个方差的事实得出（参见（Owen & Zhou，2000）或附录C.5中的引理C.1）。
Algorithm 1 requires a measure of ESS in order to evaluate the quality of a gradient estimate and, consequently, to adapt the batch size. Although several ESS measures for IS have been studied (see, e.g., (Martino et al., 2017)), to the best of our knowledge no measure specifically designed for MIS estimators has been proposed. A recent work by Elvira et al. (2018) has analyzed the classical ESS measure introduced in Section 2 and has empirically demonstrated its effectiveness in MIS. Thus, we have decided to apply it to our context as well. However, since for our application we are satisfied with a lower bound on the ESS, instead of taking the variance of the importance weights under the given proposals, we take it w.r.t. the mixture of these. This is motivated by the following proposition, which follows directly from the fact that the former variance is always smaller than the latter (see (Owen & Zhou, 2000) or Lemma C.1 in Appendix C.5).