OpenKBP: The open-access knowledge-based planning grand challenge and dataset
OpenKBP: The open-access knowledge-based planning grand challenge and dataset
复制标题
DOI：
10.1002/mp.14845
复制
发表时间：
2021-06-22
期刊：
影响因子：
3.8
通讯作者：
Chan, Timothy C. Y.
中科院分区：
文献类型：
--
作者：
Babier, Aaron;Zhang, Binghao;Chan, Timothy C. Y.
目的促进放射治疗研究中基于知识的计划（KBP）剂量预测方法的公平和一致的比较。我们举办了OpenKBP，这是2020年AAPM大挑战赛，并挑战参与者开发预测轮廓计算机断层扫描（CT）图像剂量的最佳方法。根据两个单独的评分对模型进行评价：（a）剂量评分，其评价完整的三维（3D）剂量分布，以及（B）剂量体积直方图（DVH）评分，其评价一组DVH度量。我们使用这些分数来量化基于样本外预测的模型的质量。为了开发和测试他们的模型，参与者获得了340名接受放射治疗的头颈癌患者的数据。数据分为训练（n=200），验证（n=40）和测试（n=100）数据集。所有参与者在挑战的第一（验证）阶段使用相应的数据集进行培训和验证。在第二阶段（测试）中，参与者使用他们的模型对测试数据进行量化，以量化样本外的表现，这对参与者来说是隐藏的，并用于确定最终的竞争排名。与会者还回答了一项调查，以总结他们的模式。挑战赛吸引了来自28个国家的195名参与者，其中73名参与者在验证阶段组成了44个团队，共收到1750份提交材料。测试阶段收集了来自其中28个团队的提交，代表了28种独特的预测方法。平均而言，在验证阶段，参与者将其模型的剂量和DVH评分分别提高了2.7和5.7倍。在测试阶段，第一个模型获得了最佳剂量评分（2.429）和DVH评分（1.478），这两个评分均显著优于第二个模型获得的剂量评分（2.564）和DVH评分（1.529）。最后，许多表现最好的团队报告说，他们使用了可推广的技术（例如，合奏），以实现更高的性能比他们的竞争。结论OpenKBP是基于知识的规划研究的第一个竞争。该挑战赛帮助推出了第一个平台，使研究人员能够使用大型开源数据集和标准化指标公平一致地比较KBP预测方法。OpenKBP还通过使每个人都可以访问KBP研究，使KBP研究民主化，这应该有助于加速KBP研究的进展。OpenKBP数据集是公开的，以帮助基准未来的KBP研究。
Purpose To advance fair and consistent comparisons of dose prediction methods for knowledge-based planning (KBP) in radiation therapy research. Methods We hosted OpenKBP, a 2020 AAPM Grand Challenge, and challenged participants to develop the best method for predicting the dose of contoured computed tomography (CT) images. The models were evaluated according to two separate scores: (a) dose score, which evaluates the full three-dimensional (3D) dose distributions, and (b) dose-volume histogram (DVH) score, which evaluates a set DVH metrics. We used these scores to quantify the quality of the models based on their out-of-sample predictions. To develop and test their models, participants were given the data of 340 patients who were treated for head-and-neck cancer with radiation therapy. The data were partitioned into training (n=200), validation (n=40), and testing (n=100) datasets. All participants performed training and validation with the corresponding datasets during the first (validation) phase of the Challenge. In the second (testing) phase, the participants used their model on the testing data to quantify the out-of-sample performance, which was hidden from participants and used to determine the final competition ranking. Participants also responded to a survey to summarize their models. Results The Challenge attracted 195 participants from 28 countries, and 73 of those participants formed 44 teams in the validation phase, which received a total of 1750 submissions. The testing phase garnered submissions from 28 of those teams, which represents 28 unique prediction methods. On average, over the course of the validation phase, participants improved the dose and DVH scores of their models by a factor of 2.7 and 5.7, respectively. In the testing phase one model achieved the best dose score (2.429) and DVH score (1.478), which were both significantly better than the dose score (2.564) and the DVH score (1.529) that was achieved by the runner-up models. Lastly, many of the top performing teams reported that they used generalizable techniques (e.g., ensembles) to achieve higher performance than their competition. Conclusion OpenKBP is the first competition for knowledge-based planning research. The Challenge helped launch the first platform that enables researchers to compare KBP prediction methods fairly and consistently using a large open-source dataset and standardized metrics. OpenKBP has also democratized KBP research by making it accessible to everyone, which should help accelerate the progress of KBP research. The OpenKBP datasets are available publicly to help benchmark future KBP research.