Reverse Engineering a Data Lake for Data Science******

作为一家治疗艾滋病毒、肝炎和吸毒成瘾的大型私人诊所，拉丁区城市医生诊所（“CMUQL”）设想在没有人为干预的情况下，系统地确定患者疾病的流行病学趋势。具体来说，CMUQL计划将其传统研究基础设施迁移到基于现代lambda架构的数据科学平台（DSP），先进的分布式计算技术，如MapReduce和Spark，以及数据湖。 不幸的是，这种迁移面临着许多重要的挑战，以避免最终成为“大数据墓地”。首先，面向数据湖的逆向工程仍处于起步阶段。其次，CMUQL的数据湖需要将来自第三方提供商的数据与至少两个其他数据源集成：一个十年前的手写、复制患者文档（包括医学图像）的纸质档案，格式和术语不一致;一个临时Microsoft Access/FileMaker数据库，部分基于其他两个数据源。第三，由于医疗原因，对于对（匿名）患者数据进行的任何转换、计算或分析，审计跟踪应可追溯到输入数据。** 作为迈向现代DSP的第一步，我们的目标是将当前的数据源挖掘和逆向工程到基于NoSQL的数据湖中。一名博士生和一名专业硕士生将：** 1。对数据源及其结构进行盘点 ** 2.设计基本的数据湖架构 ** 3.为现有Access数据库创建数据湖原型 ** 该项目提供的数据清单和数据湖原型将使CMUQL能够探索并开始了解威胁数千名加拿大患者的疾病。其次，该项目使2名HQP能够获得独特的研究经验，包括数据湖的逆向工程、现代数据科学架构和挖掘（非）结构化数据。

As a major private clinic treating HIV, hepatitis and drug addiction, the Clinique Médecine Urbaine of the Quartier Latin ("CMUQL"), envisions to systematically identify epidemiological trends about their patients' diseases, without human intervention. Concretely, CMUQL plans to migrate its legacy research infrastructure to a data science platform (DSP) based on a modern lambda architecture, advanced distributed computation technologies like MapReduce and Spark, and on a data lake.** Unfortunately, this migration faces a number of important challenges in order not to end up with a "big data graveyard". First, reverse engineering towards a data lake is still in its infancy. Second, CMUQL's data lake needs to integrate data from a third party provider with at least 2 other data sources: a decade-old paper archive of hand-written, copied patient documents (including medical images), with inconsistent format and terminology; and an ad hoc Microsoft Access/FileMaker database, partially based on the other two data sources. Third, due to medical reasons, for any transformation, calculation or analysis made on the (anonymized) patient data, an audit trail should be traceable back to the input data.** As a first step towards a modern DSP, we aim to mine and reverse-engineer the current data sources into a NoSQL-based data lake. One PhD student and one professional Master's student will:** 1. make an inventory of data sources and their structure** 2. design the basic data lake architecture** 3. prototype a data lake for the existing Access database** The data inventory and data lake prototype delivered by this project will enable CMUQL to explore and start understand diseases menacing thousands of Canadian patients. Second, the project allows 2 HQP to obtain unique research experience with reverse engineering towards a data lake, modern data science architectures and mining (un)structured data.******