A large language model-based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records.
A large language model-based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records.
复制标题
基于大型语言模型的生成自然语言处理框架根据临床记录进行微调，可以从电子健康记录中准确提取头痛频率。
作者：
Chiang,Chia-Chun;Luo,Man;Dumkrieger,Gina;Trivedi,Shubham;Chen,Yi-Chieh;Chao,Chieh-Ju;Schwedt,ToddJ;Sarker,Abeed;Banerjee,Imon
目的开发一种自然语言处理（Natural Language Processing，NLP）算法，能够准确地从临床病历中提取头痛频率。背景头痛频率（Headache frequency），定义为一个月（或4周）内出现头痛的天数，是评价偏头痛预防药物治疗效果的关键参数。然而，由于临床医生的文件的变化和不一致，存在重大挑战，以准确地提取头痛的频率从电子健康记录（EHR）通过传统的NLP algorithm.MethodsThis是一个回顾性的横断面研究，确定从两个三级头痛转诊中心，马约诊所亚利桑那州和马约诊所罗切斯特的患者。提取了2012年至2022年期间由15名专业临床医生（11名头痛专家和4名执业护士）撰写的所有神经病学咨询笔记，并将1915份笔记用于模型微调（90%）和测试（10%）。我们使用了四种不同的NLP框架：（1）ClinicalBERT（来自Transformers的双向编码器表示）回归模型，（2）生成式预训练的Transformer-2（GPT-2）问题筛选（QA）模型零次，（3）GPT-2 QA模型少次训练，根据临床笔记进行微调，以及（4）GPT-2生成模型少拍训练在临床笔记上进行微调，以通过考虑所包含文本的上下文来生成答案。我们的训练和测试数据集的头痛频率（标准差）分别为13.4（10.9）和14.4（11.2）。GPT-2生成模型是性能最佳的模型，准确度为0.92（0.91，0.93，95%置信区间[CI]），R2评分为0.89（0.87，0.90，95% CI），所有基于GPT-2的模型在精确匹配准确度方面均优于ClinicalBERT模型。虽然ClinicalBERT回归模型的准确率最低，为0.27（0.26，0.28），它表现出较高的R2得分为0.88（0.85，0.89），提示ClinicalBERT模型可以合理预测≤ ± 3天范围内的头痛频率，R2得分高于GPT-2 QA零激发模型或GPT-2 QA模型少激发训练微调模型。结论我们开发了一个强大的信息提取模型，基于最先进的大型语言模型，一个GPT-2生成模型，可以从EHR自由文本临床笔记中提取头痛频率，具有高准确性和R2评分。它克服了与临床医生记录头痛频率的不同方式相关的几个挑战，这些挑战是传统NLP模型不容易实现的。我们还发现，基于GPT-2的框架在从临床记录中提取头痛频率的准确性方面优于ClinicalBERT。为了促进该领域的研究，我们在GitHub上发布了GPT-2生成模型和推理代码，并获得了社区使用的开源许可。当应用于各种临床用例的不同医疗保健系统时，可能需要对算法进行额外的微调。
ObjectiveTo develop a natural language processing (NLP) algorithm that can accurately extract headache frequency from free‐text clinical notes.BackgroundHeadache frequency, defined as the number of days with any headache in a month (or 4 weeks), remains a key parameter in the evaluation of treatment response to migraine preventive medications. However, due to the variations and inconsistencies in documentation by clinicians, significant challenges exist to accurately extract headache frequency from the electronic health record (EHR) by traditional NLP algorithms.MethodsThis was a retrospective cross‐sectional study with patients identified from two tertiary headache referral centers, Mayo Clinic Arizona and Mayo Clinic Rochester. All neurology consultation notes written by 15 specialized clinicians (11 headache specialists and 4 nurse practitioners) between 2012 and 2022 were extracted and 1915 notes were used for model fine‐tuning (90%) and testing (10%). We employed four different NLP frameworks: (1) ClinicalBERT (Bidirectional Encoder Representations from Transformers) regression model, (2) Generative Pre‐Trained Transformer‐2 (GPT‐2) Question Answering (QA) model zero‐shot, (3) GPT‐2 QA model few‐shot training fine‐tuned on clinical notes, and (4) GPT‐2 generative model few‐shot training fine‐tuned on clinical notes to generate the answer by considering the context of included text.ResultsThe mean (standard deviation) headache frequency of our training and testing datasets were 13.4 (10.9) and 14.4 (11.2), respectively. The GPT‐2 generative model was the best‐performing model with an accuracy of 0.92 (0.91, 0.93, 95% confidence interval [CI]) andR2score of 0.89 (0.87, 0.90, 95% CI), and all GPT‐2–based models outperformed the ClinicalBERT model in terms of exact matching accuracy. Although the ClinicalBERT regression model had the lowest accuracy of 0.27 (0.26, 0.28), it demonstrated a highR2score of 0.88 (0.85, 0.89), suggesting the ClinicalBERT model can reasonably predict the headache frequency within a range of ≤ ± 3 days, and theR2score was higher than the GPT‐2 QA zero‐shot model or GPT‐2 QA model few‐shot training fine‐tuned model.ConclusionWe developed a robust information extraction model based on a state‐of‐the‐art large language model, a GPT‐2 generative model that can extract headache frequency from EHR free‐text clinical notes with high accuracy andR2score. It overcame several challenges related to different ways clinicians document headache frequency that were not easily achieved by traditional NLP models. We also showed that GPT‐2–based frameworks outperformed ClinicalBERT in terms of accuracy in extracting headache frequency from clinical notes. To facilitate research in the field, we released the GPT‐2 generative model and inference code with open‐source license of community use in GitHub. Additional fine‐tuning of the algorithm might be required when applied to different health‐care systems for various clinical use cases.
登录
查看更多内容
影响因子：
5
作者：
Burch, Rebecca C.;Loder, Stephen;Smitherman, Todd A.
通讯作者：
Smitherman, Todd A.
DOI：
10.1186/s10194-022-01493-x
发表时间：
2022-09-21
期刊：
The journal of headache and pain
影响因子：
--
作者：
通讯作者：
--
影响因子：
4.9
作者：
Katsuki, Masahito;Shimazu, Tomokazu;Takeshima, Takao
通讯作者：
Takeshima, Takao
影响因子：
5
作者：
Chiang, Chia-Chun;Chhabra, Nikita;Noseworthy, Peter A.
通讯作者：
Noseworthy, Peter A.
影响因子：
5
作者：
Ishii, Ryotaro;Schwedt, Todd J.;Dodick, David W.
通讯作者：
Dodick, David W.