A late-binding, distributed, NoSQL warehouse for integrating patient data from clinical trials
A late-binding, distributed, NoSQL warehouse for integrating patient data from clinical trials
复制标题
DOI：
10.1093/database/baz032
复制
发表时间：
2019-03-11
影响因子：
5.8
通讯作者：
Agrafiotis, Dimitris K.
中科院分区：
文献类型：
--
作者：
Yang, Eric;Scheff, Jeremy D.;Agrafiotis, Dimitris K.
临床试验数据通常是通过不同供应商使用不同技术和数据标准开发的多个系统收集的。这些数据需要加以整合、标准化和转换，以用于各种监测和报告目的。在需求不断变化的情况下，需要处理大量经常不一致的数据，这带来了重大的技术挑战。作为全面临床数据存储库的一部分，我们开发了一个数据仓库，该数据仓库集成了来自任何来源的患者数据，对其进行标准化，并使研究团队能够及时访问这些数据，以支持飞行中和已完成研究的各种分析任务。我们的解决方案结合了ApacheHBase、NoSQL列存储、ApachePhoenix、大规模并行关系查询引擎和用户友好的界面，利用将数据映射推迟到查询时的提取-加载-转换设计模式，促进了在不完整或不明确的规范下高效加载大量数据。这种方法允许我们维护数据的单个副本，并将其动态转换为任何所需的格式，而不需要额外的存储。可以很容易地引入对映射规范的更改，并且可以同时使用数据的多种表示。此外，通过分别对数据和转换进行版本控制，我们可以将历史地图应用于当前数据或将当前地图应用于历史数据，这简化了数据切割的维护，并促进了适应性试验的中期分析。结果是一个高度可扩展、安全和冗余的解决方案，它结合了NoSQL存储的灵活性和关系查询引擎的健壮性，以支持广泛的应用，包括临床数据管理、医疗审查、基于风险的监测、安全信号检测、已完成研究的后期分析等。
Clinical trial data are typically collected through multiple systems developed by different vendors using different technologies and data standards. That data need to be integrated, standardized and transformed for a variety of monitoring and reporting purposes. The need to process large volumes of often inconsistent data in the presence of ever-changing requirements poses a significant technical challenge. As part of a comprehensive clinical data repository, we have developed a data warehouse that integrates patient data from any source, standardizes it and makes it accessible to study teams in a timely manner to support a wide range of analytic tasks for both in-flight and completed studies. Our solution combines Apache HBase, a NoSQL column store, Apache Phoenix, a massively parallel relational query engine and a user-friendly interface to facilitate efficient loading of large volumes of data under incomplete or ambiguous specifications, utilizing an extract-load-transform design pattern that defers data mapping until query time. This approach allows us to maintain a single copy of the data and transform it dynamically into any desirable format without requiring additional storage. Changes to the mapping specifications can be easily introduced and multiple representations of the data can be made available concurrently. Further, by versioning the data and the transformations separately, we can apply historical maps to current data or current maps to historical data, which simplifies the maintenance of data cuts and facilitates interim analyses for adaptive trials. The result is a highly scalable, secure and redundant solution that combines the flexibility of a NoSQL store with the robustness of a relational query engine to support a broad range of applications, including clinical data management, medical review, risk-based monitoring, safety signal detection, post hoc analysis of completed studies and many others.