III : Medium: Collaborative Research: From Open Data to Open Data Curation

在重视机构开放性和透明度的社会趋势的推动下，开放数据的产生和共享速度超过了我们处理它的能力。许多政府和私人机构正在采用开放数据原则，即共享的数据是完整、准确和及时的。这些属性使这些数据对数据科学家、记者和公众具有重要价值。当Open Data得到有效利用时，数据科学家可以探索和分析开放资源，这反过来又允许他们调查公共政策，创造新的科学知识，并发现对社会、科学或经济计划有用的新(隐藏)价值。尽管开放数据运动已经成功地实现了让数据变得可访问的雄心，但它并没有成功地让这些有价值的数据变得易于使用。这个项目的总体目标就是解决这一缺陷。在这个项目中，我们提出了一个开放、透明和可解释的数据管理愿景。Open Data Curation使用跨数据发现、数据清理和链接以及数据集成的按需集成范例。我们的愿景是使用户能够以最少的前期工作来查询存储在数据存储库中的异类数据。用户可以在其查询中引用数据中不存在的概念和属性。按需集成系统(ODIS)通过自动确定可以转换和集成哪些数据来为所请求的概念提供数据来响应这些请求。在社会影响方面，该项目将提供算法创新，使开放数据湖上有效、直观的按需集成成为现实。我们的解决方案将使用真正的开放数据，并将对开放数据有时古怪且始终不同的特征具有健壮性。我们认为，需要深刻改变人们对数据集成和管理的看法，以推动数据科学革命，这场革命正受到不连贯的数据管理的阻碍-这项任务仍然被认为是数据科学中最耗时、最恼人、最容易出错的任务之一。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Motivated by societal trends that value institutional openness and transparency, open data is being produced and shared at a speed that surpasses our ability to process it. Many governmental and private institutions are adopting Open Data Principles that state that the shared data is complete, accurate, and timely. These properties make this data of great value to data scientists, journalists, and the public. When Open Data is used effectively, data scientists can explore and analyze open resources, which in turn allows them to investigate public policy, create new scientific knowledge, and discover new (hidden) value useful for social, scientific, or economic initiatives. Though the open data movement has succeeded in its ambition of making data accessible, it has not succeed in making this valuable data easy to use. The overarching goal of this project is to address this shortcoming.In this project, we present a vision for Open Data Curation - data curation that is open, transparent, and explainable. Open Data Curation uses an on-demand integration paradigm that spans data discovery, data cleaning and linking, and data integration. Our vision is to enable users to query heterogeneous data stored in a data repository with minimal up-front effort. Users can reference concepts and attributes in their queries that do not exist in the data. An on-demand integration system (ODIS) responds to such requests by automatically determining what data could be transformed and integrated to provide data for a requested concept. In terms of societal impact, the project will provide the algorithmic innovations to make effective, intuitive on-demand integration over open data lakes a reality. Our solutions will use real open data and will be robust to the sometimes quirky, and always diverse, characteristics of open data. We believe a profound shift in how people think about data integration and curation is needed to fuel the data science revolution which is being held back by incoherent data curation - a task that is still considered one of the most time consuming, annoying, and error-prone in data science.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.