Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
复制标题
DOI：
10.1007/s11222-016-9696-4
复制
发表时间：
2017-09-01
影响因子：
2.2
通讯作者：
Gabry, Jonah
中科院分区：
文献类型：
--
作者：
Vehtari, Aki;Gelman, Andrew;Gabry, Jonah
留一法交叉验证（LOO）和广泛适用的信息准则（WAIC）是使用在参数值的后验模拟中评估的对数似然估计拟合贝叶斯模型的逐点样本外预测准确度的方法。LOO和WAIC相对于AIC和DIC等简单的预测误差估计具有各种优势，但在实践中使用较少，因为它们涉及额外的计算步骤。在这里，我们为LOO和WAIC提供了快速稳定的计算，可以使用现有的模拟绘制来执行。我们引入了使用帕累托平滑重要性采样（PSIS）的LOO高效计算，这是一种正则化重要性权重的新过程。虽然WAIC是渐近等于LOO，我们证明了PSIS-LOO是更强大的有限的情况下，弱先验或有影响力的意见。作为我们计算的副产品，我们还获得了估计的预测误差和两个模型之间的预测误差比较的近似标准误差。我们在一个名为LOO的R包中实现了计算，并使用贝叶斯推理包Stan拟合模型进行了演示。
Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparison of predictive errors between two models. We implement the computations in an R package called loo and demonstrate using models fit with the Bayesian inference package Stan.