Enabling Data Science over Open Data and Massive Data Lakes

通过开放数据和海量数据湖实现数据科学

• 批准号: RGPIN-2018-06012
• 负责人: Miller, Renee
• 金额: $ 3.5万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2019
• 资助国家: 加拿大
• 起止时间: 2019-01-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=689675
• 关键词: Enabling; Data; Science; over; Open

In 2016, Forbes assessed that "data preparation accounts for about 80% of the work of data scientists" where preparation includes finding and collecting data, cleaning and integrating data, and managing data for data analysis. They concluded that this is also the least enjoyable part of a data scientist's job. As scientists, they would rather be deriving new knowledge and insights. The paradox is that without principled data management and preparation, those new insights are suspect at best. Data preparation and data management in support of analysis is so time consuming and unenjoyable because of the lack of tools, scientific frameworks, and mathematical foundations to support principled data preparation. I plan to devote the next five years to addressing and helping to correct this deficit. As part of my methodology, I will focus on open data, both because of its availability for scientific research and because of its importance to governments and society.******Today, most of Canada's structured open data is in CSV (comma--separated--value) files, with some in JSON, and a tiny amount in RDF. Little if any of these datasets are being published with a schema following the W3C recommendation "Data on the Web Best Practices" or other open data best practices. And the files rarely contain attribute names beyond uninformative tags (like name10). Despite the large effort in making datasets publicly available, open data publishers, like the Canadian government do not provide search functionality beyond simple keyword search on the metadata. This metadata varies greatly in quality across different datasets and publishers. Even if data publishers were to include schemas and other valuable metadata, the lack of sophisticated search functionality creates problems for data scientists who want to use open data. The heterogeneity and incompleteness of the data create new problems for understanding the true structure of the data and how it can best be aligned with other data and ultimately used effectively for data science.******My research will propose new methods for 1) finding relevant datasets (e.g., letting a data scientist find all datasets that join with hers or even all datasets that union meaningfully with hers) at interactive speeds over massive repositories of data; 2) new methods and mathematical foundations for the discovery of structure over open data (e.g., in CSV files that contain misaligned or pivoted data); and 3) new methods for aligning open data and data within massive public or private data lakes. I plan to make the methods I develop open source along with benchmarks for helping other scientists to develop and evaluate data preparation, collection and management solutions for massive open data. This work will extend the societal reach of Open Data at all levels (federal, provincial, and municipal), allowing this valuable data to be used more easily, in more effective and principled ways, for more purposes.**

2016 年，《福布斯》评估“数据准备约占数据科学家工作的 80%”，其中准备包括查找和收集数据、清理和集成数据以及管理数据以进行数据分析。他们的结论是，这也是数据科学家工作中最不愉快的部分。作为科学家，他们宁愿获得新的知识和见解。矛盾的是，如果没有原则性的数据管理和准备，这些新见解充其量也是值得怀疑的。由于缺乏支持原则性数据准备的工具、科学框架和数学基础，支持分析的数据准备和数据管理非常耗时且令人不愉快。我计划在接下来的五年里致力于解决和帮助纠正这一缺陷。作为我的方法论的一部分，我将重点关注开放数据，既因为它可用于科学研究，也因为它对政府和社会的重要性。******如今，加拿大的大部分结构化开放数据都采用 CSV（逗号分隔值）文件格式，其中一些采用 JSON 格式，还有少量采用 RDF 格式。 这些数据集几乎没有按照 W3C 建议的“网络数据最佳实践”或其他开放数据最佳实践的模式进行发布。这些文件很少包含除了无信息标签（如 name10）之外的属性名称。尽管在公开数据集方面付出了巨大的努力，但开放数据发布者（例如加拿大政府）除了对元数据进行简单的关键字搜索之外，不提供搜索功能。不同数据集和发布者的元数据质量差异很大。即使数据发布者要包含模式和其他有价值的元数据，缺乏复杂的搜索功能也会给想要使用开放数据的数据科学家带来问题。 数据的异质性和不完整性为理解数据的真实结构以及如何最好地与其他数据对齐并最终有效地用于数据科学带来了新的问题。 *****我的研究将提出新方法：1）在海量数据存储库上以交互速度查找相关数据集（例如，让数据科学家找到与她的数据集相结合的所有数据集，甚至与她的数据集有意义地结合的所有数据集）； 2）用于发现开放数据结构的新方法和数学基础（例如，在包含未对齐或旋转数据的 CSV 文件中）； 3）调整开放数据和大规模公共或私人数据湖中的数据的新方法。我计划将我开发的方法以及基准开源，以帮助其他科学家开发和评估海量开放数据的数据准备、收集和管理解决方案。这项工作将扩大开放数据在各级（联邦、省和市）的社会影响力，使这些有价值的数据能够更轻松地、以更有效和更有原则的方式用于更多目的。**