Redundancy in electronic health record corpora: analysis, impact on text mining performance and mitigation strategies.
Redundancy in electronic health record corpora: analysis, impact on text mining performance and mitigation strategies.
复制标题
DOI：
10.1186/1471-2105-14-10
复制
发表时间：
2013-01-16
影响因子：
3
通讯作者：
Elhadad N
中科院分区：
文献类型：
--
作者：
Cohen R;Elhadad M;Elhadad N
电子健康记录（EHR）数据和特别是自由文本患者记录的可用性不断增加，为表型提取提供了机会。特别是文本挖掘方法可以通过将命名实体提到映射到术语和聚类语义相关术语来帮助疾病建模。然而，电子病历语料库，表现出特定的统计和语言特征相比，在生物医学文献领域的语料库。我们专注于复制和粘贴冗余：临床医生通常复制和粘贴信息，从以前的笔记时，记录当前的病人遇到。因此，在纵向患者记录中，人们期望观察到大量冗余。在本文中，我们提出了三个研究问题：（一）如何在大规模文本语料库中量化冗余？(ii)传统观点认为，较大的语料库在文本挖掘中会产生更好的结果。但是，观察到的EHR冗余如何影响文本挖掘？这种冗余是否会引入一种偏差，从而扭曲学习模型？或者冗余通过突出语料库中稳定和重要的子集带来了好处？(iii)如何减轻冗余对文本挖掘的影响？我们分析了一个大规模的电子病历语料库和量化冗余的词和语义概念的重复。我们观察到约30%的冗余水平和非标准分布的单词和概念。我们衡量冗余的影响，两个标准的文本挖掘应用程序：搭配识别和主题建模。我们比较这些方法的结果与控制水平的冗余合成数据，并观察到显着的性能变化。最后，我们比较了两种缓解策略，以避免冗余引起的偏见：（i）基线策略，仅保留语料库中每个患者的最后一个音符;（ii）使用高效的基于指纹的算法删除冗余音符。a对于文本挖掘，使用指纹对EHR语料库进行预处理会产生更好的结果。在应用文本挖掘技术之前，必须仔细注意所分析的语料库的结构。虽然数据清理对于低级文本特征（例如，编码和拼写），高级和难以量化的语料库特征，如自然发生的冗余，也会损害文本挖掘。指纹技术使文本挖掘技术能够利用EHR语料库中的可用数据，同时避免冗余引入的偏见。
The increasing availability of Electronic Health Record (EHR) data and specifically free-text patient notes presents opportunities for phenotype extraction. Text-mining methods in particular can help disease modeling by mapping named-entities mentions to terminologies and clustering semantically related terms. EHR corpora, however, exhibit specific statistical and linguistic characteristics when compared with corpora in the biomedical literature domain. We focus on copy-and-paste redundancy: clinicians typically copy and paste information from previous notes when documenting a current patient encounter. Thus, within a longitudinal patient record, one expects to observe heavy redundancy. In this paper, we ask three research questions: (i) How can redundancy be quantified in large-scale text corpora? (ii) Conventional wisdom is that larger corpora yield better results in text mining. But how does the observed EHR redundancy affect text mining? Does such redundancy introduce a bias that distorts learned models? Or does the redundancy introduce benefits by highlighting stable and important subsets of the corpus? (iii) How can one mitigate the impact of redundancy on text mining? We analyze a large-scale EHR corpus and quantify redundancy both in terms of word and semantic concept repetition. We observe redundancy levels of about 30% and non-standard distribution of both words and concepts. We measure the impact of redundancy on two standard text-mining applications: collocation identification and topic modeling. We compare the results of these methods on synthetic data with controlled levels of redundancy and observe significant performance variation. Finally, we compare two mitigation strategies to avoid redundancy-induced bias: (i) a baseline strategy, keeping only the last note for each patient in the corpus; (ii) removing redundant notes with an efficient fingerprinting-based algorithm. aFor text mining, preprocessing the EHR corpus with fingerprinting yields significantly better results. Before applying text-mining techniques, one must pay careful attention to the structure of the analyzed corpora. While the importance of data cleaning has been known for low-level text characteristics (e.g., encoding and spelling), high-level and difficult-to-quantify corpus characteristics, such as naturally occurring redundancy, can also hurt text mining. Fingerprinting enables text-mining techniques to leverage available data in the EHR corpus, while avoiding the bias introduced by redundancy.
登录
查看更多内容
影响因子：
7.5
作者：
Dredze, Mark;Kulesza, Alex;Crammer, Koby
通讯作者：
Crammer, Koby
DOI：
10.1136/jamia.1994.95236146
发表时间：
1994-03-01
影响因子：
6.4
作者：
FRIEDMAN, C;ALDERSON, PO;JOHNSON, SB
通讯作者：
JOHNSON, SB
DOI：
10.1016/s1386-5056(02)00053-9
发表时间：
2002-12-04
影响因子：
4.9
作者：
Hahn, U;Romacker, M;Schulz, S
通讯作者：
Schulz, S
影响因子：
2.5
作者：
Cormode, Graham;Hadjieleftheriou, Marios
通讯作者：
Hadjieleftheriou, Marios
影响因子：
6
作者：
Blei, DM;Ng, AY;Jordan, MI
通讯作者：
Jordan, MI