Bayesian regression tree ensembles that adapt to smoothness and sparsity
Bayesian regression tree ensembles that adapt to smoothness and sparsity
复制标题
DOI：
10.1111/rssb.12293
复制
发表时间：
2017-07
期刊：
影响因子：
--
通讯作者：
A. Linero;Yun Yang
中科院分区：
文献类型：
--
作者：
A. Linero;Yun Yang
决策树集成是获得回归函数的灵活估计的有用工具。这些方法的例子包括梯度增强决策树、随机森林以及贝叶斯分类和回归树。树丛的两个潜在缺点是缺乏流畅性和易受维度诅咒的影响。我们表明，这些问题可以通过考虑稀疏诱导软决策树来克服，在软决策树中决策被视为概率决策。我们在贝叶斯加性回归树框架的背景下实现了这一方法，并通过在基准数据集上的测试来说明其良好的性能。我们证明了在高维情形下，对于稀疏函数和具有加性结构的函数，后验分布集中在极小极大速率(直到对数因子)，其中协变量空间的维度被允许在样本大小中近乎指数地增长，从而为我们的方法提供了强有力的理论支持。我们的方法还可以适应未知的光滑度和稀疏度，并且可以通过对现有的贝叶斯加性回归树算法进行最小修改来实现。
Ensembles of decision trees are a useful tool for obtaining flexible estimates of regression functions. Examples of these methods include gradient‐boosted decision trees, random forests and Bayesian classification and regression trees. Two potential shortcomings of tree ensembles are their lack of smoothness and their vulnerability to the curse of dimensionality. We show that these issues can be overcome by instead considering sparsity inducing soft decision trees in which the decisions are treated as probabilistic. We implement this in the context of the Bayesian additive regression trees framework and illustrate its promising performance through testing on benchmark data sets. We provide strong theoretical support for our methodology by showing that the posterior distribution concentrates at the minimax rate (up to a logarithmic factor) for sparse functions and functions with additive structures in the high dimensional regime where the dimensionality of the covariate space is allowed to grow nearly exponentially in the sample size. Our method also adapts to the unknown smoothness and sparsity levels, and can be implemented by making minimal modifications to existing Bayesian additive regression tree algorithms.