Enabling Data Science over Open Data and Massive Data Lakes

点击翻译按钮获取中文摘要

In 2016, Forbes assessed that "data preparation accounts for about 80% of the work of data scientists" where preparation includes finding and collecting data, cleaning and integrating data, and managing data for data analysis. They concluded that this is also the least enjoyable part of a data scientist's job. As scientists, they would rather be deriving new knowledge and insights. The paradox is that without principled data management and preparation, those new insights are suspect at best. Data preparation and data management in support of analysis is so time consuming and unenjoyable because of the lack of tools, scientific frameworks, and mathematical foundations to support principled data preparation. I plan to devote the next five years to addressing and helping to correct this deficit. As part of my methodology, I will focus on open data, both because of its availability for scientific research and because of its importance to governments and society.Today, most of Canada's structured open data is in CSV (comma--separated--value) files, with some in JSON, and a tiny amount in RDF. Little if any of these datasets are being published with a schema following the W3C recommendation "Data on the Web Best Practices" or other open data best practices. And the files rarely contain attribute names beyond uninformative tags (like name10). Despite the large effort in making datasets publicly available, open data publishers, like the Canadian government do not provide search functionality beyond simple keyword search on the metadata. This metadata varies greatly in quality across different datasets and publishers. Even if data publishers were to include schemas and other valuable metadata, the lack of sophisticated search functionality creates problems for data scientists who want to use open data. The heterogeneity and incompleteness of the data create new problems for understanding the true structure of the data and how it can best be aligned with other data and ultimately used effectively for data science.My research will propose new methods for 1) finding relevant datasets (e.g., letting a data scientist find all datasets that join with hers or even all datasets that union meaningfully with hers) at interactive speeds over massive repositories of data; 2) new methods and mathematical foundations for the discovery of structure over open data (e.g., in CSV files that contain misaligned or pivoted data); and 3) new methods for aligning open data and data within massive public or private data lakes. I plan to make the methods I develop open source along with benchmarks for helping other scientists to develop and evaluate data preparation, collection and management solutions for massive open data. This work will extend the societal reach of Open Data at all levels (federal, provincial, and municipal), allowing this valuable data to be used more easily, in more effective and principled ways, for more purposes.