RAPID: Time-Sensitive Human Forest and Model Forecasts for COVID-19 Vaccine and Treatment Trials

在快速蔓延的流行病期间，准确、有时间特异性的预测对于规划和决策非常重要。特别是，有效的COVID-19疫苗是否能在9个月、12个月或18个月内上市，是一个事关国家利益的问题。该项目的主要目标是比较基于人群的特定时间结果预测新方法的准确性-例如COVID-19治疗和疫苗的临床试验过渡-与新机器学习模型的准确性。该研究将检查人群和建模方法的相对优势，并探索两者在预测临床试验结果方面的组合。预测比赛是该项目收集人类数据的主要方法。它从2020年开始，一直持续到2021年。鼓励对预测和临床试验感兴趣的人报名参加，无论他们的背景如何。研究参与者完成调查和预测培训，然后将有机会在几个月内对特定试验事件进行概率预测，并定期提供准确性反馈。为了扩大这项工作的影响，研究团队通过公共卫生信息渠道传播有关COVID-19治疗和疫苗临床试验阶段过渡的综合预测。这些预测，结合提供给研究参与者的预测性培训和准确性反馈，可能有助于协调公共卫生和临床开发工作，以克服大流行病。该项目的主要研究目标是提高基于人群的方法，机器模型和两者的集成的预测性能。心理学家已经证明，通过在历史参考类的背景下检查预测问题，采取外部观点可以提高准确性。基于人群的方法，被称为人类森林，结合参考类预测和集体智慧的方法，从一组预测人员中产生数据驱动的估计。特定时间的人类森林变量采用生存分析方法，使预测者能够构建参考类，并在缺失数据的情况下获得无偏的历史估计。在决策科学方面，研究目标包括测试具有历史估计的界面的影响;理解参考类选择的心理学;检查判断预测的时间范围敏感性;以及评估主题专业知识与一般预测能力的相对重要性。在机器建模方面，研究目标包括将生存型模型集成到机器学习中，并使用双层优化来选择超参数以提高其性能。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Accurate, time-specific predictions are important for planning and decision making during fast-moving pandemics. In particular, whether an effective COVID-19 vaccine will be available in 9, 12, or 18 months is an issue of vital national interest. The main objective of this project is to compare the accuracy of a new method for crowd-based forecasting of time-specific outcomes–such as clinical trial transitions of COVID-19 treatments and vaccines–to that of new machine learning models. The research will examine the relative strengths of crowd and modeling methods and explore combinations of the two in predicting clinical trial results. A forecasting tournament is the project’s main method for human data collection. It starts in 2020 and continues until 2021. People with interest in forecasting and clinical trials are encouraged to sign up for participation, independently of their background. Study participants complete surveys and forecasting training, and will then have the opportunity to make probabilistic forecasts on specific trial events over several months, with regular accuracy feedback. To broaden the impacts of this work, the research team disseminates the aggregate forecasts about clinical trial phase transition of COVID-19 treatments and vaccines through public health information channels. These forecasts, combined with predictive training and accuracy feedback provided to study participants, may aid the coordination of public health and clinical development efforts to overcome the pandemic. The primary research goal of the project is to improve the predictive performance of crowd-based methods, machine models and ensembles of the two. Psychologists have shown that taking the outside view, by examining a prediction problem in context of historical reference classes, improves accuracy. The crowd-based approach, referred to as human forest, combines reference class forecasting and collective intelligence approaches to produce data-driven estimates from a group of forecasters. The time-specific human forest variant employs a survival analysis approach, enabling forecasters to construct reference classes and obtain unbiased historical estimates in the presence of missing data. On the decision science front, the research goals include testing the effects of interfaces featuring historical estimates; understanding the psychology of reference class selection; examining time-scope sensitivity in judgmental forecasting; and assessing the relative importance of subject matter expertise versus general predictive competence. On the machine-modeling front, the research goals include integrating survival-type models into machine learning and improving their performance using bi-level optimization to choose hyper-parameters. The results are released as soon as they become available.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.