A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis.
A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis.
复制标题
DOI：
10.1001/jamanetworkopen.2022.27423
复制
发表时间：
2022-08-01
影响因子：
13.8
通讯作者：
Bridges, S. Louis, Jr.
中科院分区：
文献类型：
--
作者：
Sun, Dongmei;Nguyen, Thanh M.;Allaway, Robert J.;Wang, Jelai;Chung, Verena;Yu, Thomas, V;Mason, Michael;Dimitrovsky, Isaac;Ericson, Lars;Li, Hongyang;Guan, Yuanfang;Israel, Ariel;Olar, Alex;Pataki, Balint Armin;Stolovitzky, Gustavo;Guinney, Justin;Gulko, Percio S.;Frazier, Mason B.;Chen, Jake Y.;Costello, James C.;Bridges, S. Louis, Jr.
全球范围内的合作能否开发机器学习算法来自动量化关节间隙狭窄和侵蚀，以改进目前类风湿关节炎（RA）放射摄影的视觉检查方法？这项预后研究评估了一项国际众包竞赛，使用来自3项RA患者临床研究的手/手腕和脚的评分x线片来开发机器学习算法来量化RA的损害。通过挑战后独立验证数据集验证了所提交算法的准确性和可重复性。这些发现表明，在更大的队列中进行改进和验证后，这些算法单独或组合可以纳入电子健康记录，有助于更明智和更精确地管理RA。这项诊断/预后研究评估了提交给国际众包竞赛的机器学习算法，以开发自动量化类风湿性关节炎（RA）患者损伤的放射学证据的方法。需要一种自动化、准确的方法来进行无偏评估，量化临床试验中手、手腕和脚的放射图像上关节间隙狭窄和侵蚀的累积情况，监测关节损伤的时间，协助风湿病学家做出治疗决定。这种方法有可能直接集成到电子健康记录中。设计并实施一项国际众包竞赛，以促进机器学习方法的发展，以量化类风湿关节炎（RA）的放射损伤。本诊断/预后研究描述了类风湿关节炎逆向工程评估和方法2-对话（RA2-DREAM挑战），该研究使用现有的影像学图像和专家整理的来自2项临床研究的Sharp-van - der - Heijde （SvH）评分（来自562名患者的674组影像学）进行训练（367组）、排行榜（119组）和最终评估（188组）。挑战参与者的任务是开发自动量化整体损伤（子挑战1）、关节空间缩小（子挑战2）和侵蚀（子挑战3）的方法。这项挑战于2020年6月30日结束。将提交的算法得出的分数与专家策划的SvH分数进行比较，并创建基线模型进行基准比较。使用加权均方根误差（RMSE）对性能进行排名。利用自举数据中的贝叶斯因子对各算法的性能和再现性进行了评估，并利用挑战后独立验证数据集进一步进行了评估。RA2-DREAM挑战赛共收到了来自7个国家的26个参与者或团队的173份参赛作品，其中13份作品被纳入最终评估。加权均方根误差指标显示，获胜算法产生的分数非常接近专家策划的SvH分数。顶级团队包括子挑战1的石林队（加权均方根误差为0.44），子挑战2的HYL-YFG（加权均方根误差为0.38）和子挑战3的Gold Therapy（加权均方根误差为0.43）。Bootstrapping/Bayes因子法和挑战后独立验证证实了最终评估与挑战后独立验证数据集的再现性和估计一致性指数，子挑战1为0.71，子挑战2为0.78，子挑战3为0.82。RA2-DREAM挑战导致了算法的发展，提供了可行、快速和准确的方法来量化RA的关节损伤。最终，这些方法可以帮助RA关节损伤的研究，并可以整合到电子健康记录中，通过提供及时、可靠和定量的信息来制定治疗决策，帮助临床医生更好地为患者服务，以防止进一步的损伤。
Can a worldwide collaborative effort develop machine learning algorithms to quantify joint space narrowing and erosions automatically to improve the current visual inspection approach to radiography in rheumatoid arthritis (RA)? This prognostic study assesses an international, crowdsourcing competition using scored radiographs of hands/wrists and feet from 3 clinical studies of patients with RA to develop machine learning algorithms to quantify damage in RA. The accuracy and reproducibility of the submitted algorithms were confirmed with a postchallenge independent validation data set. These findings suggest that after refining and validating with larger cohorts, these algorithms alone or in combination could be incorporated into electronic health records, contributing to more informed and precise management of RA. This diagnostic/prognostic study presents assesses machine learning algorithms submitted to an international crowdsourcing contest to develop methos to automatically quantify radiographic evidence of damage in patients with rheumatoid arthritis (RA). An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images of the hands and wrists, and feet for clinical trials, monitoring of joint damage over time, assisting rheumatologists with treatment decisions. Such a method has the potential to be directly integrated into electronic health records. To design and implement an international crowdsourcing competition to catalyze the development of machine learning methods to quantify radiographic damage in rheumatoid arthritis (RA). This diagnostic/prognostic study describes the Rheumatoid Arthritis 2–Dialogue for Reverse Engineering Assessment and Methods (RA2-DREAM Challenge), which used existing radiographic images and expert-curated Sharp-van der Heijde (SvH) scores from 2 clinical studies (674 radiographic sets from 562 patients) for training (367 sets), leaderboard (119 sets), and final evaluation (188 sets). Challenge participants were tasked with developing methods to automatically quantify overall damage (subchallenge 1), joint space narrowing (subchallenge 2), and erosions (subchallenge 3). The challenge was finished on June 30, 2020. Scores derived from submitted algorithms were compared with the expert-curated SvH scores, and a baseline model was created for benchmark comparison. Performances were ranked using weighted root mean square error (RMSE). The performance and reproductivity of each algorithm was assessed using Bayes factor from bootstrapped data, and further evaluated with a postchallenge independent validation data set. The RA2-DREAM Challenge received a total of 173 submissions from 26 participants or teams in 7 countries for the leaderboard round, and 13 submissions were included in the final evaluation. The weighted RMSEs metric showed that the winning algorithms produced scores that were very close to the expert-curated SvH scores. Top teams included Team Shirin for subchallenge 1 (weighted RMSE, 0.44), HYL-YFG (Hongyang Li and Yuanfang Guan) subchallenge 2 (weighted RMSE, 0.38), and Gold Therapy for subchallenge 3 (weighted RMSE, 0.43). Bootstrapping/Bayes factor approach and the postchallenge independent validation confirmed the reproducibility and the estimation concordance indices between final evaluation and postchallenge independent validation data set were 0.71 for subchallenge 1, 0.78 for subchallenge 2, and 0.82 for subchallenge 3. The RA2-DREAM Challenge resulted in the development of algorithms that provide feasible, quick, and accurate methods to quantify joint damage in RA. Ultimately, these methods could help research studies on RA joint damage and may be integrated into electronic health records to help clinicians serve patients better by providing timely, reliable, and quantitative information for making treatment decisions to prevent further damage.
登录
查看更多内容
影响因子：
48
作者：
Marbach, Daniel;Costello, James C.;Kueffner, Robert;Vega, Nicole M.;Prill, Robert J.;Camacho, Diogo M.;Allison, Kyle R.;Kellis, Manolis;Collins, James J.;Stolovitzky, Gustavo
通讯作者：
Stolovitzky, Gustavo
影响因子：
4.7
作者：
Bridges, S. Louis, Jr.;Causey, Zenoria L.;van der Heijde, Desiree M.
通讯作者：
van der Heijde, Desiree M.
影响因子：
27.4
作者：
Cross, Marita;Smith, Emma;March, Lyn
通讯作者：
March, Lyn
影响因子：
51.1
作者：
Guinney, Justin;Wang, Tao;Laajala, Teemu D.;Winner, Kimberly Kanigel;Bare, J. Christopher;Neto, Elias Chaibub;Khan, Suleiman A.;Peddinti, Gopal;Airola, Antti;Pahikkala, Tapio;Mirtti, Tuomas;Yu, Thomas;Bot, Brian M.;Shen, Liji;Abdallah, Kald;Norman, Thea;Friend, Stephen;Stolovitzky, Gustavo;Soule, Howard;Sweeney, Christopher J.;Ryan, Charles J.;Scher, Howard I.;Sartor, Oliver;Xie, Yang;Aittokallio, Tero;Zhou, Fang Liz;Costello, James C.;Abdallah, Kald;Aittokallio, Tero;Airola, Antti;Anghel, Catalina;Azima, Helia;Baertsch, Robert;Ballester, Pedro J.;Bare, Chris;Bhandari, Vinayak;Bot, Brian M.;Dang, Cuong C.;Dunba, Maria Bekker-Nielsen;Buchardt, Ann-Sophie;Buturovic, Ljubomir;Cao, Da;Chalise, Prabhakar;Cho, Junwoo;Chu, Tzu-Ming;Coley, R. Yates;Conjeti, Sailesh;Correia, Sara;Costello, James C.;Dai, Ziwei;Dai, Junqiang;Dargatz, Philip;Delavarkhan, Sam;Deng, Detian;Dhanik, Ankur;Du, Yu;Elangovan, Aparna;Ellis, Shellie;Elo, Laura L.;Espiritu, Shadrielle M.;Fan, Fan;Farshi, Ashkan B.;Freitas, Ana;Fridley, Brooke;Friend, Stephen;Fuchs, Christiane;Gofer, Eyal;Peddinti, Gopalacharyulu;Graw, Stefan;Greiner, Russ;Guan, Yuanfang;Guinney, Justin;Guo, Jing;Gupta, Pankaj;Guyer, Anna I.;Han, Jiawei;Hansen, Niels R.;Chang, Billy H. W.;Hirvonen, Outi;Huang, Barbara;Huang, Chao;Hwang, Jinseub;Ibrahim, Joseph G.;Jayaswal, Vivek;Jeon, Jouhyun;Ji, Zhicheng;Juvvadi, Deekshith;Jyrkkio, Sirkku;Kanigel-Winner, Kimberly;Katouzian, Amin;Kazanov, Marat D.;Khan, Suleiman A.;Khayyer, Shahin;Kim, Dalho;Golinska, Agnieszka K.;Koestler, Devin;Kokowicz, Fernanda;Kondofersky, Ivan;Krautenbacher, Norbert;Krstajic, Damjan;Kumar, Luke;Kurz, Christoph;Kyan, Matthew;Laajala, Teemu D.;Laimighofer, Michael;Lee, Eunjee;Lesinski, Wojciech;Li, Miaozhu;Li, Ye;Lian, Qiuyu;Liang, Xiaotao;Lim, Minseong;Lin, Henry;Lin, Xihui;Lu, Jing;Mahmoudian, Mehrad;Manshaei, Roozbeh;Meier, Richard;Miljkovic, Dejan;Mirtti, Tuomas;Mnich, Krzysztof;Navab, Nassir;Neto, Elias C.;Newton, Yulia;Norman, Thea;Pahikkala, Tapio;Pal, Subhabrata;Park, Byeongju;Patel, Jaykumar;Pathak, Swetabh;Pattin, Alejandrina;Ankerst, Donna P.;Peng, Jian;Petersen, Anne H.;Philip, Robin;Piccolo, Stephen R.;Poelsterl, Sebastian;Polewko-Klim, Aneta;Rao, Karthik;Ren, Xiang;Rocha, Miguel;Rudnicki, Witold R.;Ryan, Charles J.;Ryu, Hyunnam;Sartor, Oliver;Scherb, Hagen;Sehgal, Raghav;Seyednasrollah, Fatemeh;Shang, Jingbo;Shao, Bin;Shen, Liji;Sher, Howard;Shiga, Motoki;Sokolov, Artem;Soellner, Julia F.;Song, Lei;Soule, Howard;Stolovitzky, Gustavo;Stuart, Josh;Sun, Ren;Sweeney, Christopher J.;Tahmasebi, Nazanin;Tan, Kar-Tong;Tomaziu, Lisbeth;Usset, Joseph;Vang, Yeeleng S.;Vega, Roberto;Vieira, Vitor;Wang, David;Wang, Difei;Wang, Junmei;Wang, Lichao;Wang, Sheng;Wang, Tao;Wang, Yue;Wolfinger, Russ;Wong, Chris;Wu, Zhenke;Xiao, Jinfeng;Xie, Xiaohui;Xie, Yang;Xin, Doris;Yang, Hojin;Yu, Nancy;Yu, Thomas;Yu, Xiang;Zahedi, Sulmaz;Zanin, Massimiliano;Zhang, Chihao;Zhang, Jingwen;Zhang, Shihua;Zhang, Yanchun;Zhou, Fang Liz;Zhu, Hongtu;Zhu, Shanfeng;Zhu, Yuxin
通讯作者：
Zhu, Yuxin
影响因子：
6.7
作者：
Costello, J. C.;Stolovitzky, G.
通讯作者：
Stolovitzky, G.