III: Small: Semantic Version Management in Data Lakes

数据为我们的经济提供动力。那些从事数据工作的人努力寻找有效的方法来提取知识并处理不断增长的数据量。这种增长不仅以新的数据源为特征，而且以数据复制为特征——即复制、集成和修改数据集，从而创建新版本的数据集。值得注意的是，国际数据公司（International Data Corporation）等咨询公司估计，商业中使用的大多数新生成的数据都是现有数据的版本。因此，理解数据需要从语义上理解数据版本控制，这成为处理和管理数据的关键因素。该项目将专注于推进对数据版本控制的科学理解，并将从根本上为使用数据的任何科学或活动做出贡献，如今数据涵盖了所有人类活动的大量内容。为了推进开放数据科学，该项目将为数据变化的语义理解奠定基础，从而产生新版本的数据，并将引入可扩展的工具来发现和解释数据变化。这将有助于开发有效的框架来处理数据科学管道中的多个数据版本，以及设计包含和管理数据复制的系统。这项工作也有望通过促进负责任和开放的数据科学来造福社会。它的解决方案将向公众开放，并与高质量的经过精心策划的基准一起提供，这些基准本身将具有科学价值，可以进行比较和解决科学辩论，从而推动这一重要领域的发展。该项目还将使用“负责任的数据科学”的各个方面，旨在确保处理数据时的公平性、准确性、保密性和透明度。这个项目将开发一种新的范式，我们称之为语义版本管理。其愿景是使用户能够以最少的前期工作来理解通常驻留在数据湖中的大量版本。该项目的主要目标是使目前主要依赖文件名的数据科学家能够找到数据集的“正确”版本，以查看和理解数据集之间所做的更改（清理、值输入、集成等）。研究方法建立，集成和扩展可扩展的数据发现工作；程序算例与数据变换综合；从不一致和不完整的证据中学习图式映射。本项目将开发支持数据版本控制语义理解的方法，为研究数据版本奠定基础，并建立评估和基准化数据版本控制的新方法。具体来说，该项目将解决以下基础研究挑战：1)恢复对数据进行的转换，并解释一个数据集与另一个数据集版本的差异；2)有效地从海量表存储库或数据湖中查找数据集的版本；3)了解版本集合中的版本历史，并构建一个图表来表达数据版本创建背后的故事。在整个开发过程中，该项目还将开发新的评估框架，不仅考虑解决方案的正确性，而且考虑其可解释性。语义版本管理的一个重要动机是让用户对他们正在使用的数据更加信任。如果他们了解用于从一个版本派生另一个版本的转换，他们就可以更好地理解一个版本是否满足他们的数据科学任务的需要。此外，将生成新的基准并与社区共享，以鼓励开放科学，并允许与新的或替代的版本理解方法进行可靠的比较。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data fuels our economy. Those who work with data invest their efforts in finding effective ways to extract knowledge and to handle its increasing size. This growth is not only characterized by new sources of data, but also by data replication - that is, the copying, integration, and modification of datasets that creates new versions of datasets. Notably, consultants such as the International Data Corporation estimate that most of the newly generated data being used in business are versions of existing data. Understanding data therefore requires a semantic understanding of data versioning, which becomes a key ingredient in handling and managing data. This project will focus on advancing the scientific understanding of data versioning and will fundamentally contribute to any science or activity that uses data, which nowadays covers a tremendous amount of all human activity. To advance open data science, this project will lay the foundations for semantic understanding of data changes that result in new versions of data and will introduce scalable tools to uncover and explain data changes. This will contribute both to the development of effective frameworks to handle multiple data versions within a data science pipeline as well as to the design of systems that incorporate and manage data replication. This work is expected to also benefit society by facilitating responsible and open data science. Its solutions will be made publicly available and provided alongside high-quality highly curated benchmarks that themselves will have scientific value in allowing comparisons and settling scientific debates in order to advance this important field. The project will also use aspects of "responsible data science", aiming to ensure fairness, accuracy, confidentiality, and transparency when working with data. This project will develop a new paradigm we call semantic version management. The vision is to enable users, with minimal upfront effort, to understand the multitude of versions that typically reside in data lakes. The main objective of the project is to enable data scientists who currently rely mainly on file names to find the "right" version of a dataset to see and understand the changes (cleaning, value imputation, integration, and others) that have been made between datasets. The research methodology builds-on, integrates, and extends work on scalable data discovery; program by example and data transformation synthesis; and learning schema mappings from inconsistent and incomplete evidence. This project will develop methods to support the semantic understanding of data versioning, lay the foundations for studying data versions, and establish new methods for evaluating and benchmarking data versioning. Specifically, this project will address the following fundamental research challenges: 1) recovering transformations done to data and explain how one dataset differs from another version of the dataset; 2) efficiently finding versions of a dataset from within a massive table repository or data lake; and 3) understanding the version history among a collection of versions and constructing a graph that expresses the story behind the creation of the data versions. Throughout the development, this project will also develop new evaluation frameworks that not only consider the correctness of solutions, but also their explainability. An important motivation for semantic version management is to give users more trust in the data they are using. If they understand the transformations used to derive one version from another, they can better understand if a version meets the needs of their data science task. In addition, new benchmarks will be generated and shared with the community to encourage open science and allow reliable comparison with new or alternative approaches to version understanding.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.