Human Forests versus Random Forest Models in Prediction

准确的预测是不确定条件下有效决策的关键。心理学研究表明，可以通过考虑外部观点来提高预测性判断：将问题置于类似历史案例的背景下，而不是专注于其独特性。但选择正确的比较是困难的：统计学家至少从19世纪就开始研究所谓的参考阶级问题。该项目的主要目标是评估相对于新的和已建立的机器学习模型，用于众包参考类判断和产生概率预测的新方法的性能。这种方法被称为“人类森林”，通过使预测者能够从历史案例数据库中构建参考类，促进了外部视角思维。人类森林方法与随机森林机器模型有一个概念上的联系。在这两种情况下，预测都是基于分类树中评估的频率。虽然随机森林模型使用训练数据来构建树木，但人类森林依赖于预测者的集体知识。该项目将检查两种方法的相对优势，并探索两者的结合。我们还将评估提高个别预报员准确性的方法。这一建议的知识价值在于它承诺通过集体智慧来解决参考阶级问题。该项目将比较辅以元认知训练和统计聚合技术的人类森林与随机森林模型和人机混合方法的准确性。后者将使用双级优化，在机器学习中使用优化提供了一个进步，目的是推动机器学习和人类能力的前沿。核心随机实验将集中于临床试验预测，即预测癌症治疗进展的概率。研究方法将采用自然的、纵向的、大规模的在线实验，并将比较主题专家和通才的表现。该项目还将为研究人员和学生提供机器学习和集体智能方面的培训，并为高中STEM课程、本科和研究生统计学和决策课程的互动练习开发材料。评估一般预测技能与主题专业知识的相对重要性可能有助于解决完全依赖专家的领域的技能短缺问题。这项研究旨在提高临床试验结果和类似复杂活动的可预测性。对临床试验项目成功的准确预测可能反过来改善风险管理、资源分配，并最终导致更广泛地获得挽救生命的治疗。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Accurate predictions are key to effective decision making under uncertainty. Psychology research has shown that predictive judgments can be improved by considering the outside view: placing a problem in the context of similar historical cases, rather than focusing on its unique features. But choosing the right comparison is difficult: statisticians have studied the so-called reference class problem since at least the 19th century. The main objective of this project is to assess the performance of a new method for crowdsourcing reference-class judgments and producing probability forecasts, relative to new and established machine learning models. The method, called human forests, promotes outside-view thinking by enabling forecasters to construct reference classes from a database of historical cases. The human forests method shares a conceptual connection with random forest machine models. In both, predictions are based on frequencies assessed in classification trees. While random forest models use training data to build the trees, human forests rely on forecaster' collective knowledge. The project will examine the relative strengths of both methods and explore combinations of the two. We will also assess methods for improving the accuracy of individual forecasters. The intellectual merit of the proposal resides in its promise to address the reference class problem through collective intelligence. The project will compare the accuracy of human forests, complemented with metacognitive training and statistical aggregation techniques, with that of random forest models, and a human-machine hybrid approach. The latter will use bi-level optimization, providing an advancement in the use of optimization in machine learning, with the aim of pushing the frontier of both machine learning and human capabilities. The core randomized experiments will focus on clinical trial forecasting, namely, predicting the probability of advancement for cancer treatments. The study methods will utilize naturalistic, longitudinal, large-scale online experiments, and will compare the performance of subject-matter experts and generalists. The project will also provide training for researchers and students in machine learning and collective intelligence and develop materials for interactive exercises in high-school STEM classes, undergraduate and graduate courses in statistics and decision making. Assessing the relative importance of general forecasting skill versus subject matter expertise may help address skill scarcity problems in areas dependent exclusively on specialists. The research aims to improve the predictability of clinical trial outcomes and similarly complex activities. Accurate forecasts regarding the success of clinical trial programs may in turn improve risk management, resource allocation, and ultimately result in wider availability of life-saving treatments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.