RAPID: Time-Sensitive Human Forest and Model Forecasts for COVID-19 Vaccine and Treatment Trials

准确、具体时间的预测对于在快速发展的大流行期间进行规划和决策非常重要。特别是，9个月、12个月或18个月后能否获得有效的新冠肺炎疫苗是一个事关国家利益的问题。该项目的主要目标是比较一种基于人群预测特定时间结果的新方法的准确性-例如新冠肺炎治疗和疫苗的临床试验过渡-与新机器学习模型的准确性。这项研究将检验人群和建模方法的相对优势，并探索两者在预测临床试验结果方面的组合。预报锦标赛是该项目收集人类数据的主要方法。它从2020年开始，一直持续到2021年。鼓励对预测和临床试验感兴趣的人报名参加，与他们的背景无关。研究参与者完成调查和预测培训，然后将有机会在几个月内对特定试验事件做出概率预测，并定期提供准确的反馈。为了扩大这项工作的影响，研究团队通过公共卫生信息渠道传播对新冠肺炎治疗和疫苗临床试验阶段转变的总体预测。这些预测与向研究参与者提供的预测性培训和准确反馈相结合，可能有助于协调公共卫生和临床开发工作，以克服大流行。该项目的主要研究目标是提高基于人群的方法、机器模型和两者的集成的预测性能。心理学家已经证明，从外部看问题，通过在历史参考课的背景下检查预测问题，可以提高准确性。这种基于人群的方法被称为人类森林，它结合了参考类别预测和集体智能方法，从一组预测者那里产生数据驱动的估计。特定时间的人类森林变种采用生存分析方法，使预测者能够构建参考类别，并在存在缺失数据的情况下获得无偏见的历史估计。在决策科学方面，研究目标包括测试以历史估计为特色的界面的效果；了解参考类别选择的心理；检查判断预测中的时间范围敏感性；以及评估主题专业知识与一般预测能力的相对重要性。在机器建模方面，研究目标包括将生存型模型集成到机器学习中，并通过双层优化来选择超参数来提高其性能。结果一有结果就会公布。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，认为值得支持。

Accurate, time-specific predictions are important for planning and decision making during fast-moving pandemics. In particular, whether an effective COVID-19 vaccine will be available in 9, 12, or 18 months is an issue of vital national interest. The main objective of this project is to compare the accuracy of a new method for crowd-based forecasting of time-specific outcomes–such as clinical trial transitions of COVID-19 treatments and vaccines–to that of new machine learning models. The research will examine the relative strengths of crowd and modeling methods and explore combinations of the two in predicting clinical trial results. A forecasting tournament is the project’s main method for human data collection. It starts in 2020 and continues until 2021. People with interest in forecasting and clinical trials are encouraged to sign up for participation, independently of their background. Study participants complete surveys and forecasting training, and will then have the opportunity to make probabilistic forecasts on specific trial events over several months, with regular accuracy feedback. To broaden the impacts of this work, the research team disseminates the aggregate forecasts about clinical trial phase transition of COVID-19 treatments and vaccines through public health information channels. These forecasts, combined with predictive training and accuracy feedback provided to study participants, may aid the coordination of public health and clinical development efforts to overcome the pandemic. The primary research goal of the project is to improve the predictive performance of crowd-based methods, machine models and ensembles of the two. Psychologists have shown that taking the outside view, by examining a prediction problem in context of historical reference classes, improves accuracy. The crowd-based approach, referred to as human forest, combines reference class forecasting and collective intelligence approaches to produce data-driven estimates from a group of forecasters. The time-specific human forest variant employs a survival analysis approach, enabling forecasters to construct reference classes and obtain unbiased historical estimates in the presence of missing data. On the decision science front, the research goals include testing the effects of interfaces featuring historical estimates; understanding the psychology of reference class selection; examining time-scope sensitivity in judgmental forecasting; and assessing the relative importance of subject matter expertise versus general predictive competence. On the machine-modeling front, the research goals include integrating survival-type models into machine learning and improving their performance using bi-level optimization to choose hyper-parameters. The results are released as soon as they become available.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.