Trees, forests, and impurity-based variable importance in regression
Trees, forests, and impurity-based variable importance in regression
复制标题
回归中的树木、森林和基于杂质的变量重要性
DOI：
--
复制
发表时间：
2020
影响因子：
1.5
通讯作者：
Erwan Scornet
中科院分区：
文献类型：
--
作者：
Erwan Scornet
树集成方法（如随机森林[Breiman，2001]）在处理高维表格数据集方面非常流行，特别是因为它们具有良好的预测精度。然而，当机器学习用于决策问题时，解决最佳预测程序可能是不合理的，因为明智的决策需要深入理解算法预测过程。不幸的是，随机森林本质上是不可解释的，因为它们的预测结果来自对数百棵决策树的平均。一个经典的方法来获得知识，这个所谓的黑箱算法是计算变量的重要性，这是用来评估每个输入变量的预测影响。变量重要性然后被用来对变量进行排序或选择，因此在数据分析中发挥了重要作用。然而，以这种方式使用随机森林变量重要性是没有道理的：我们甚至不知道这些量估计的是什么。在本文中，我们分析了两个著名的随机森林变量的重要性，平均减少杂质（MDI）。我们证明，如果输入变量是独立的，在没有相互作用，MDI提供了一个方差分解的输出，其中每个变量的贡献是明确的。我们还研究模型之间的依赖输入变量或相互作用，变量的重要性本质上是不明确的。我们的分析表明，与单一树木相比，使用森林可能存在一些好处。
Tree ensemble methods such as random forests [Breiman, 2001] are very popular to handle high-dimensional tabular data sets, notably because of their good predictive accuracy. However, when machine learning is used for decision-making problems, settling for the best predictive procedures may not be reasonable since enlightened decisions require an in-depth comprehension of the algorithm prediction process. Unfortunately, random forests are not intrinsically interpretable since their prediction results from averaging several hundreds of decision trees. A classic approach to gain knowledge on this so-called black-box algorithm is to compute variable importances, that are employed to assess the predictive impact of each input variable. Variable importances are then used to rank or select variables and thus play a great role in data analysis. Nevertheless, there is no justification to use random forest variable importances in such way: we do not even know what these quantities estimate. In this paper, we analyze one of the two well-known random forest variable importances, the Mean Decrease Impurity (MDI). We prove that if input variables are independent and in absence of interactions, MDI provides a variance decomposition of the output, where the contribution of each variable is clearly identified. We also study models exhibiting dependence between input variables or interaction, for which the variable importance is intrinsically ill-defined. Our analysis shows that there may exist some benefits to use a forest compared to a single tree.