Black-Box Science: Ideas and Insights for Learning-Based Statistical Inference

该项目旨在开发必要的工具，使科学家能够在实际科学环境中更好地利用机器学习的全部力量。在当前的大数据时代，机器学习算法作为建模复杂系统和预测未来结果的优秀、准确的工具而脱颖而出。事实证明，确定这些特定算法实际起作用的原因以及这些预测是如何产生的是一个更大的挑战，但理解这些方面对于实际的科学应用至关重要。例如，如果一种算法预测你有患某种特定疾病的风险，你会本能地不太关心你感染这种疾病的确切几率，而更关心为什么你更有可能感染这种疾病，以及你是否可以采取措施防止感染。这项工作将开发工具，使科学家能够更容易地确定哪些变量对算法的性能影响最大，以及其他一些变量集合是否可能为预测的结果提供另一种但同样准确的解释。还将从数学上探索这些算法的各个组成部分，以确定其中一些是否可以被借用并插入到更简单的模型中，以便获得不仅更准确，而且更容易解释的预测。这个项目寻求在机器学习的背景下开发有效的统计推理手段，特别是强调随机森林。具体地说，将开发一种计算效率高的假设检验，允许以与原始算法类似的工作来计算特征重要性的p值。除了这些测试外，还将制定一个框架，用于描述模型选择过程本身的不确定性，以便不仅提供对所获得的最佳模型的见解，而且还说明可能存在多少具有类似预测能力的替代模型。最后，将对随机性在监督学习集成中的基础作用进行深入研究。关于这种随机性有益影响的经验教训将被用来在适当的环境中提高更传统模型的性能。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project seeks to develop essential tools that will allow scientists to better harness the full power of machine learning in practical scientific settings. In the current era of big data, machine learning algorithms have set themselves apart as excellent, accurate tools for modeling complex systems and predicting future outcomes. Determining why those particular algorithms actually work and how those predictions are generated has proven to be a much greater challenge, yet understanding these aspects is crucial for practical scientific use. For example, if an algorithm predicts that you are at risk for a particular disease, you will instinctively care less about the exact percentage chance you have of getting it and much more about why you are more likely to get it and whether there is something you could do to prevent getting it. This work will develop tools that allow scientists to more easily determine which variables most affect an algorithm's performance and whether some other collection of variables might offer an alternative but equally accurate explanation for the outcomes predicted. Various components of these algorithms will also be explored mathematically to determine whether some of them can be borrowed and inserted into simpler models in order to obtain predictions that are not only more accurate, but are also more easily explainable.This project seeks to develop efficient means of statistical inference within a machine learning context with an emphasis on random forests in particular. Specifically, a computationally efficient hypothesis test will be developed that allows for p-values for feature importance to be calculated with similar effort to the original algorithm. In addition to these tests, a framework for characterizing the uncertainty in the model selection process itself will be developed to provide insights into not just the optimal model obtained, but also to illustrate how many alternative models may exist with similar predictive power. Finally, an in-depth study on the fundamental role of randomness in supervised learning ensembles will be undertaken. Lessons learned about the helpful effects of such randomness will be utilized to boost performance of more traditional models in appropriate settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.