Investigating the generalizability of natural language processing of EMR data

描述（由申请人提供）： 电子病历（EMR）为提高医疗质量提供了令人印象深刻的机会，但挑战阻碍了这一愿景的实现。例如，易于用于分析的编码EMR数据通常是不完整的（由于EMR实现中自由文本临床笔记的流行），并且由于标准词汇表和系统实现的差异，来自不同EMR的数据通常是不相称的。虽然信息学研究已经表明，使用自然语言处理（NLP）对临床文本的特定方面进行自动编码是可行的，但将这些信息学发展转化为大规模护理质量评估仍然存在挑战。迄今为止，成功的自动化质量评估NLP解决方案往往是特定于（a）目标问题或临床焦点，（B）EMR数据系统，以及（c）实施NLP解决方案的人员或团队的应用程序。在这项研究中，我们建议开始通过开发，评估和免费提供一个通用的NLP开发工具套件来解决实施团队的特殊性问题。这些工具将使NLP系统能够广泛采用，以从自由文本临床笔记中提取和编码数据。知识编辑工具包将通过帮助用户定义构成特定领域知识模块的规则、概念和术语来简化特定问题知识的开发，从而允许任何信息学家开发NLP应用程序。NLP应用程序验证工具包将允许根据来自任何EMR的独立编码测试记录的黄金标准对应用程序进行快速测试和评估。为了评估工具包对NLP泛化能力的影响，我们将让三名临床信息学家每人构建两个NLP应用程序（总共六个不同的应用程序）。他们的应用之一将识别一系列常见的临床体征或症状（例如，"持续性咳嗽"），它们是相对离散的概念，使用简单的语言术语用于许多不同的临床目的。他们的第二个应用程序将评估行为咨询（例如，“酒精咨询”），其使用复杂的语言结构用于专门的临床目的。我们将描述和评估的准确性的解决方案对独立编码的测试集的医疗记录。我们将量化和比较创建这些解决方案的难度，这些难度是通过构建应用程序所需的时间、迭代次数以及所采用的概念和规则的数量来衡量的，我们还将分析所创建的解决方案的内容和准确性的变化。此外，我们将使用定性技术来评估使用开发工具的易用性;学习工具的难度;以及遇到的特定类型的问题、限制和错误。这样的NLP开发工具套件有可能允许简单，优雅和可靠的NLP解决方案，无论临床问题领域或开发解决方案的人如何。

DESCRIPTION (provided by applicant): The electronic medical record (EMR) offers impressive opportunities for increasing care quality, but challenges stand in the way of realizing this vision. For example, coded EMR data readily available for analysis typically are incomplete (due to the prevalence of free-text clinical notes in EMR implementations), and data from different EMRs are often incommensurate due to differences in standard vocabularies and system implementations. While informatics research has shown the feasibility of automatically coding specific aspects of clinical text using Natural Language Processing (NLP), challenges remain for translating these informatics developments into large-scale care quality assessments. To date, successful NLP solutions for automated quality assessment have tended to be applications that are specific to (a) the target problem or clinical focus, (b) the EMR data system, and (c) the person or team that implements the NLP solution. In this study, we propose to begin addressing the problem of implementation team specificity by developing, evaluating, and making freely available a generalizable NLP development tool suite. The tools will enable widespread adoption of NLP systems to extract and code data from free text clinical notes. The Knowledge Editing Toolkit will simplify development of problem-specific knowledge by helping the user define the rules, concepts, and terms that constitute a domain-specific knowledge module, thus allowing any informaticist to develop an NLP application. The NLP Application Validation Toolkit will allow rapid testing and evaluation of the application against a gold standard of independently-coded test records from any EMR. To evaluate the effects of the toolkits on NLP generalizability, we will have three clinical informaticists each build two NLP applications (for a total of six distinct applications). One of their applications will identify a constellation of common clinical signs or symptoms (e.g., "persistent cough") that are relatively discrete concepts using simple language terms for many different clinical purposes. Their second application will assess behavioral counseling (e.g., "alcohol counseling"), which uses complex language constructs for dedicated clinical purposes. We will describe and evaluate the accuracy of the solutions against independently coded test sets of medical records. We will quantify and compare the difficulty of creating these solutions as measured by the time, number of iterations required to build the applications, and the number of concepts and rules employed, as well as analyze variability in content and accuracy of the solutions created. In addition, we will use qualitative techniques to assess the ease of using the development tools; the difficulty in learning the tools; and specific types of problems, limitations, and bugs encountered. Such an NLP development tool suite has the potential to allow simple, elegant, and reliably good NLP solutions regardless of the clinical problem domain or the person developing the solution.