Robust modal regression with direct log-density derivative estimation
Robust modal regression with direct log-density derivative estimation
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Hiroaki Sasaki;Tomoya Sakai;T. Kanamori
中科院分区：
文献类型：
--
作者：
Hiroaki Sasaki;Tomoya Sakai;T. Kanamori
模态回归旨在估计给定输入变量的输出变量的条件密度函数的全局模态（即全局最大值），并导致回归方法对重尾或偏态噪声具有鲁棒性。条件模态通常通过求模态回归风险（MRR）的最大化来估计。为了将梯度法应用于最大化，最根本的挑战是精确逼近MRR的梯度，而不是MRR本身。为了克服这一挑战，本文采用了一种直接逼近MRR梯度的新方法。为了近似梯度，我们开发了最小二乘对数密度导数估计器的核化和基于神经网络的版本，它直接近似对数密度的导数，而不需要密度估计。在直接逼近MRR梯度的基础上，提出了一种带核的模态回归方法，并推导了一种新的基于不动点法的参数更新规则。然后，从理论上证明了所导出的更新规则对条件模态具有单调爬坡性。此外，我们指出我们直接逼近梯度的方法与最近的复杂随机梯度方法（例如Adam）兼容，然后提出了另一种基于神经网络的模态回归方法。最后，在各种人工数据集和基准数据集上验证了所提方法的优越性能。
Modal regression is aimed at estimating the global mode (i.e., global maximum) of the conditional density function of the output variable given input variables, and has led to regression methods robust against heavy-tailed or skewed noises. The conditional mode is often estimated through maximization of the modal regression risk (MRR). In order to apply a gradient method for the maximization, the fundamental challenge is accurate approximation of the gradient of MRR, not MRR itself. To overcome this challenge, in this paper, we take a novel approach of directly approximating the gradient of MRR. To approximate the gradient, we develop kernelized and neural-network-based versions of the least-squares log-density derivative estimator, which directly approximates the derivative of the log-density without density estimation. With direct approximation of the MRR gradient, we first propose a modal regression method with kernels, and derive a new parameter update rule based on a fixed-point method. Then, the derived update rule is theoretically proved to have a monotonic hill-climbing property towards the conditional mode. Furthermore, we indicate that our approach of directly approximating the gradient is compatible with recent sophisticated stochastic gradient methods (e.g., Adam), and then propose another modal regression method based on neural networks. Finally, the superior performance of the proposed methods is demonstrated on various artificial and benchmark datasets.