Learning Universal Patient Representations with Hierarchical Transformers

项目摘要 电子健康记录包含大量有关患者健康状态的信息，可用于挖掘 多种目的，包括临床研究和改善护理点的决策。这 信息可以表示为结构化变量、非结构化文本和图像等数据 类型。在这项工作中，我们开发了新的模型来表示非结构化文本，这些模型利用 强大的神经模型被称为预先训练的变压器。我们建议让这些模型在很大程度上可用 通过添加层次结构层来对较小文本块的摘要进行操作并缩小文本，从而使文本变长 在较小区块上运行的编码器的大小。首先，我们开发了一个较小的句子和视频编码器 段落大小的文本，通过使用一种名为极端蒸馏的技术，该技术从 较大型号的产品。我们还建议预先训练文本的层次模型，方法是利用 较小的编码器喜欢从第一个目标。我们利用公共和私有数据集， 尝试不同的培训前任务和架构。我们的最终目标是将 从文本中学习的表示法，以及从结构化数据和图像等更成熟的领域中学习的表示法。我们 设计实验，回答如何最好地合并这些不同的信息源的问题，以及 将它们应用于可能需要多个信息源的重要临床分类用例 以获得准确的性能。具体地说，我们解决了在紧急情况下预测损伤严重程度的临床任务。 科室，并预测重症监护病房患者的诊断和预后。

Project Summary Electronic health records contain a wealth of information about patient health status that can be mined for multiple purposes, including clinical research and improved decision-making at the point of care. This information can be represented as structured variables, unstructured text, and images, among other data types. In this work, we develop new models for representing the unstructured text that take advantage of powerful neural models called pre-trained transformers. We propose to make these models usable for much longer texts by adding hierarchical layers to operate over summaries of smaller chunks of text, and shrinking the size of the encoder that operates on smaller chunks. First, we develop a smaller encoder for sentence and paragraph-sized texts, by using a technique called extreme distillation that trains smaller models from the output of larger models. We also propose to pre-train hierarchical models for text, by taking advantage of smaller encoders like that from the first aim. We take advantage of both public and private datasets and experiment with different pre-training tasks and architectures. Our final aim proposes to combine representations learned from text with those from the more mature areas of structured data and images. We design experiments that answer the question of how best to merge these different information sources, and apply them to important clinical classification use cases that are likely to require multiple information sources for accurate performance. Specifically, we address the clinical tasks of predicting injury severity in emergency departments, and predicting diagnosis and prognosis of patients in intensive care units.