Human Forests versus Random Forest Models in Prediction

准确的预测是在不确定情况下做出有效决策的关键。心理学研究表明，预测性判断可以通过考虑外部观点来改善：将问题置于类似历史案例的背景下，而不是关注其独特的特征。但选择正确的比较是困难的：统计学家至少从19世纪世纪就开始研究所谓的参照类问题。该项目的主要目标是评估一种新的众包参考类判断和概率预测方法的性能，相对于新的和已建立的机器学习模型。这种方法被称为人类森林，通过使预测者能够从历史案例数据库中构建参考类来促进外部视角思维。人类森林方法与随机森林机器模型有着概念上的联系。在这两种情况下，预测都是基于分类树中评估的频率。虽然随机森林模型使用训练数据来构建树木，但人类森林依赖于预测员的集体知识。该项目将研究这两种方法的相对优势，并探索两者的结合。我们还将评估提高个人预报准确性的方法。该提案的学术价值在于它承诺通过集体智慧解决参考类问题。该项目将比较人类森林的准确性，辅之以元认知训练和统计汇总技术，以及随机森林模型和人机混合方法的准确性。后者将使用双层优化，在机器学习中使用优化方面取得进步，旨在推动机器学习和人类能力的前沿。核心随机实验将侧重于临床试验预测，即预测癌症治疗的进展概率。研究方法将利用自然，纵向，大规模的在线实验，并将比较主题专家和通才的表现。该项目还将为研究人员和学生提供机器学习和集体智慧方面的培训，并为高中STEM课程、本科和研究生统计和决策课程的互动练习开发材料。评估一般预测技能相对于主题专门知识的相对重要性，可能有助于解决完全依赖专家的领域的技能短缺问题。该研究旨在提高临床试验结果和类似复杂活动的可预测性。对临床试验项目成功与否的准确预测，反过来可能会改善风险管理、资源分配，并最终导致更广泛的救生治疗的可用性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Accurate predictions are key to effective decision making under uncertainty. Psychology research has shown that predictive judgments can be improved by considering the outside view: placing a problem in the context of similar historical cases, rather than focusing on its unique features. But choosing the right comparison is difficult: statisticians have studied the so-called reference class problem since at least the 19th century. The main objective of this project is to assess the performance of a new method for crowdsourcing reference-class judgments and producing probability forecasts, relative to new and established machine learning models. The method, called human forests, promotes outside-view thinking by enabling forecasters to construct reference classes from a database of historical cases. The human forests method shares a conceptual connection with random forest machine models. In both, predictions are based on frequencies assessed in classification trees. While random forest models use training data to build the trees, human forests rely on forecaster' collective knowledge. The project will examine the relative strengths of both methods and explore combinations of the two. We will also assess methods for improving the accuracy of individual forecasters. The intellectual merit of the proposal resides in its promise to address the reference class problem through collective intelligence. The project will compare the accuracy of human forests, complemented with metacognitive training and statistical aggregation techniques, with that of random forest models, and a human-machine hybrid approach. The latter will use bi-level optimization, providing an advancement in the use of optimization in machine learning, with the aim of pushing the frontier of both machine learning and human capabilities. The core randomized experiments will focus on clinical trial forecasting, namely, predicting the probability of advancement for cancer treatments. The study methods will utilize naturalistic, longitudinal, large-scale online experiments, and will compare the performance of subject-matter experts and generalists. The project will also provide training for researchers and students in machine learning and collective intelligence and develop materials for interactive exercises in high-school STEM classes, undergraduate and graduate courses in statistics and decision making. Assessing the relative importance of general forecasting skill versus subject matter expertise may help address skill scarcity problems in areas dependent exclusively on specialists. The research aims to improve the predictability of clinical trial outcomes and similarly complex activities. Accurate forecasts regarding the success of clinical trial programs may in turn improve risk management, resource allocation, and ultimately result in wider availability of life-saving treatments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.