High-dimensional penalty selection via minimum description length principle
High-dimensional penalty selection via minimum description length principle
复制标题
通过最小描述长度原则进行高维惩罚选择
DOI：
10.1007/s10994-018-5732-2
复制
发表时间：
2018
期刊：
影响因子：
7.5
通讯作者：
Yamanishi Kenji
中科院分区：
文献类型：
--
作者：
Miyaguchi Kohei;Yamanishi Kenji
基于最小描述长度（MDL）原则，我们解决了正则化的惩罚选择问题。特别是，我们认为惩罚函数的设计空间是高维的。在这种情况下，幸运归一化最大似然（LNML）最小化方法是有利的，因为LNML根据MDL原则量化了具有任何形式的惩罚函数的正则化模型的良好性，并引导我们通过高维空间找到好的惩罚函数。然而，LNML的最小化需要两个主要的挑战：（1）LNML的归一化因子的计算和（2）它在高维空间中的最小化。本文提出了一种新的正则化选择方法（MDL-RS），在该方法中，LNML（uLNML）的紧上界被最小化，并保证局部收敛。我们的主要贡献是推导uLNML，这是一个统一的间隙上界的LNML的解析表达式。这以近似的方式解决了上述挑战，因为它允许我们精确地近似LNML，然后有效地最小化它。实验结果表明，MDL-RS提高了正则化估计的泛化性能，特别是当模型具有冗余参数时。
We tackle the problem of penalty selection for regularization on the basis of the minimum description length (MDL) principle. In particular, we consider that the design space of the penalty function is high-dimensional. In this situation, the luckiness-normalized-maximum-likelihood (LNML)-minimization approach is favorable, because LNML quantifies the goodness of regularized models with any forms of penalty functions in view of the MDL principle, and guides us to a good penalty function through the high-dimensional space. However, the minimization of LNML entails two major challenges: (1) the computation of the normalizing factor of LNML and (2) its minimization in high-dimensional spaces. In this paper, we present a novel regularization selection method (MDL-RS), in which a tight upper bound of LNML (uLNML) is minimized with local convergence guarantee. Our main contribution is the derivation of uLNML, which is a uniform-gap upper bound of LNML in an analytic expression. This solves the above challenges in an approximate manner because it allows us to accurately approximate LNML and then efficiently minimize it. The experimental results show that MDL-RS improves the generalization performance of regularized estimates specifically when the model has redundant parameters.