Black-Box Science: Ideas and Insights for Learning-Based Statistical Inference

该项目旨在开发必要的工具，使科学家能够在实际的科学环境中更好地利用机器学习的全部力量。在当前的大数据时代，机器学习算法已经成为建模复杂系统和预测未来结果的优秀、准确的工具。事实证明，确定这些特定算法实际上为什么有效以及这些预测是如何产生的是一个更大的挑战，但理解这些方面对于实际的科学应用至关重要。例如，如果一个算法预测你有患某种疾病的风险，你会本能地不关心你得到它的确切百分比的机会，而更多地关心为什么你更有可能得到它，以及你是否可以做些什么来防止得到它。这项工作将开发工具，使科学家能够更容易地确定哪些变量最影响算法的性能，以及是否其他一些变量的集合可能为预测的结果提供另一种但同样准确的解释。这些算法的各个组成部分也将在数学上进行探索，以确定其中一些是否可以借用并插入到更简单的模型中，以获得不仅更准确，而且更容易解释的预测。该项目旨在在机器学习背景下开发有效的统计推断方法，特别强调随机森林。具体而言，将开发一种计算效率高的假设检验，该检验允许以与原始算法类似的努力计算特征重要性的p值。除了这些测试之外，还将开发一个用于表征模型选择过程本身的不确定性的框架，不仅可以深入了解所获得的最佳模型，还可以说明可能存在多少具有类似预测能力的替代模型。最后，将深入研究随机性在监督学习集成中的基本作用。关于这种随机性的有益影响的经验教训将被用来提高更传统的模型在适当的设置中的性能。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

This project seeks to develop essential tools that will allow scientists to better harness the full power of machine learning in practical scientific settings. In the current era of big data, machine learning algorithms have set themselves apart as excellent, accurate tools for modeling complex systems and predicting future outcomes. Determining why those particular algorithms actually work and how those predictions are generated has proven to be a much greater challenge, yet understanding these aspects is crucial for practical scientific use. For example, if an algorithm predicts that you are at risk for a particular disease, you will instinctively care less about the exact percentage chance you have of getting it and much more about why you are more likely to get it and whether there is something you could do to prevent getting it. This work will develop tools that allow scientists to more easily determine which variables most affect an algorithm's performance and whether some other collection of variables might offer an alternative but equally accurate explanation for the outcomes predicted. Various components of these algorithms will also be explored mathematically to determine whether some of them can be borrowed and inserted into simpler models in order to obtain predictions that are not only more accurate, but are also more easily explainable.This project seeks to develop efficient means of statistical inference within a machine learning context with an emphasis on random forests in particular. Specifically, a computationally efficient hypothesis test will be developed that allows for p-values for feature importance to be calculated with similar effort to the original algorithm. In addition to these tests, a framework for characterizing the uncertainty in the model selection process itself will be developed to provide insights into not just the optimal model obtained, but also to illustrate how many alternative models may exist with similar predictive power. Finally, an in-depth study on the fundamental role of randomness in supervised learning ensembles will be undertaken. Lessons learned about the helpful effects of such randomness will be utilized to boost performance of more traditional models in appropriate settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.