The 2019 n2c2/OHNLP Track on Clinical Semantic Textual Similarity: Overview.
The 2019 n2c2/OHNLP Track on Clinical Semantic Textual Similarity: Overview.
复制标题
2019年n2 c2/OHNLP临床语义文本相似性跟踪：概述。
DOI：
10.2196/23375
复制
发表时间：
2020-11-27
影响因子：
3.2
通讯作者：
Liu H
中科院分区：
文献类型：
--
作者：
Wang Y;Fu S;Shen F;Henry S;Uzuner O;Liu H
语义文本相似性是一般英语领域中的一项常见任务，用于评估两个文本片段的底层语义彼此等效的程度。临床语义文本相似度（ClinicalSTS）是临床领域的语义文本相似度任务，试图测量两个临床文本片段之间的语义等价程度。由于电子病历系统中模板的频繁使用，临床记录中存在大量冗余文本，这使得ClinicalSTS对于下游临床自然语言处理应用中临床文本的二次利用至关重要，例如临床文本摘要、临床语义提取、临床信息检索等。我们的目标是发布 ClinicalSTS 数据集并激励自然语言处理和生物医学信息学社区解决临床领域的语义文本相似性任务。我们于 2018 年组织了第一个 BioCreative/OHNLP ClinicalSTS 共享任务，提供了真实世界的 ClinicalSTS 数据集。我们在 2019 年继续与国家 NLP 临床挑战赛 (n2c2) 和开放健康自然语言处理 (OHNLP) 联盟合作共同完成任务，并组织了 2019 年 n2c2/OHNLP ClinicalSTS 赛道。我们发布了一个更大的 ClinicalSTS 数据集，包含 1642 个临床句子对，其中包括来自 2018 年共享任务的 1068 对和来自 2 个电子健康记录系统 GE 和 Epic 的 1006 个新对。我们向参与团队发布了 80% (1642/2054) 的数据来开发和微调语义文本相似性系统，并使用剩余的 20% (412/2054) 作为盲测来评估他们的系统。本次研讨会与美国医学信息学协会2019年年会同期举行。在签署 n2c2/OHNLP ClinicalSTS 共享任务的 78 个国际团队中，有 33 个团队总共提交了 87 份有效系统提交材料。排名前 3 的系统由 IBM 研究中心、国家生物技术信息中心和佛罗里达大学生成，皮尔逊相关性分别为 r=.9010、r=.8967 和 r=.8864。大多数性能最佳的系统都使用最先进的神经语言模型（例如 BERT 和 XLNet）以及深度学习中最先进的训练模式（例如预训练和微调模式以及多任务学习）。总体而言，尽管大部分训练数据是 GE 句子对，但参与系统在 Epic 句子对上的表现优于 GE 句子对。 2019 年 n2c2/OHNLP ClinicalSTS 共享任务侧重于计算现实世界中临床笔记生成的临床文本句子的语义相似度。吸引了大批国际团队。 ClinicalSTS 共享任务可以继续作为自然语言处理和医学信息学社区研究人员开发和改进临床文本语义文本相似性技术的场所。
Semantic textual similarity is a common task in the general English domain to assess the degree to which the underlying semantics of 2 text segments are equivalent to each other. Clinical Semantic Textual Similarity (ClinicalSTS) is the semantic textual similarity task in the clinical domain that attempts to measure the degree of semantic equivalence between 2 snippets of clinical text. Due to the frequent use of templates in the Electronic Health Record system, a large amount of redundant text exists in clinical notes, making ClinicalSTS crucial for the secondary use of clinical text in downstream clinical natural language processing applications, such as clinical text summarization, clinical semantics extraction, and clinical information retrieval. Our objective was to release ClinicalSTS data sets and to motivate natural language processing and biomedical informatics communities to tackle semantic text similarity tasks in the clinical domain. We organized the first BioCreative/OHNLP ClinicalSTS shared task in 2018 by making available a real-world ClinicalSTS data set. We continued the shared task in 2019 in collaboration with National NLP Clinical Challenges (n2c2) and the Open Health Natural Language Processing (OHNLP) consortium and organized the 2019 n2c2/OHNLP ClinicalSTS track. We released a larger ClinicalSTS data set comprising 1642 clinical sentence pairs, including 1068 pairs from the 2018 shared task and 1006 new pairs from 2 electronic health record systems, GE and Epic. We released 80% (1642/2054) of the data to participating teams to develop and fine-tune the semantic textual similarity systems and used the remaining 20% (412/2054) as blind testing to evaluate their systems. The workshop was held in conjunction with the American Medical Informatics Association 2019 Annual Symposium. Of the 78 international teams that signed on to the n2c2/OHNLP ClinicalSTS shared task, 33 produced a total of 87 valid system submissions. The top 3 systems were generated by IBM Research, the National Center for Biotechnology Information, and the University of Florida, with Pearson correlations of r=.9010, r=.8967, and r=.8864, respectively. Most top-performing systems used state-of-the-art neural language models, such as BERT and XLNet, and state-of-the-art training schemas in deep learning, such as pretraining and fine-tuning schema, and multitask learning. Overall, the participating systems performed better on the Epic sentence pairs than on the GE sentence pairs, despite a much larger portion of the training data being GE sentence pairs. The 2019 n2c2/OHNLP ClinicalSTS shared task focused on computing semantic similarity for clinical text sentences generated from clinical notes in the real world. It attracted a large number of international teams. The ClinicalSTS shared task could continue to serve as a venue for researchers in natural language processing and medical informatics communities to develop and improve semantic textual similarity techniques for clinical text.