Reverse Engineering a Data Lake for Data Science******

点击翻译按钮获取中文摘要

As a major private clinic treating HIV, hepatitis and drug addiction, the Clinique Médecine Urbaine of the Quartier Latin ("CMUQL"), envisions to systematically identify epidemiological trends about their patients' diseases, without human intervention. Concretely, CMUQL plans to migrate its legacy research infrastructure to a data science platform (DSP) based on a modern lambda architecture, advanced distributed computation technologies like MapReduce and Spark, and on a data lake.** Unfortunately, this migration faces a number of important challenges in order not to end up with a "big data graveyard". First, reverse engineering towards a data lake is still in its infancy. Second, CMUQL's data lake needs to integrate data from a third party provider with at least 2 other data sources: a decade-old paper archive of hand-written, copied patient documents (including medical images), with inconsistent format and terminology; and an ad hoc Microsoft Access/FileMaker database, partially based on the other two data sources. Third, due to medical reasons, for any transformation, calculation or analysis made on the (anonymized) patient data, an audit trail should be traceable back to the input data.** As a first step towards a modern DSP, we aim to mine and reverse-engineer the current data sources into a NoSQL-based data lake. One PhD student and one professional Master's student will:** 1. make an inventory of data sources and their structure** 2. design the basic data lake architecture** 3. prototype a data lake for the existing Access database** The data inventory and data lake prototype delivered by this project will enable CMUQL to explore and start understand diseases menacing thousands of Canadian patients. Second, the project allows 2 HQP to obtain unique research experience with reverse engineering towards a data lake, modern data science architectures and mining (un)structured data.******