III: Medium: Dataset Search and Ranking for Data Augmentation and Explanation

正在收集和编目的关于环境、社会和人口的数据量呈爆炸式增长。此外，随着对透明和开放数据的推动，科学家、政府和组织越来越多地在Web上提供这些数据。与分析和机器学习的进步相结合，这种越来越多的数据获取从理论上应该允许在世界上许多最重要的科学和社会问题上取得进展。然而，由于一个主要的技术障碍，这个机会经常被错过：目前，领域专家几乎不可能从大量公开可用的信息中筛选出其特定应用所需的数据集。数据存储库平台，如CKAN和Datverse，以及数据集搜索引擎，如Google DataSet Search，旨在使共享和查找数据集变得更容易。但这些系统只支持简单的基于关键字的查询和元数据搜索，这不足以让用户适当地指定他们的信息需求。研究人员设想了一种新型的数据集搜索引擎，通过支持一组更丰富的可发现性查询来释放开放数据中尚未开发的价值，以迎合分析任务的需求，并帮助构建和改进机器学习模型。通过赋予科学家和从业者发现相关数据的能力，该项目具有巨大的潜力，可以刺激领域内和跨领域的数据重复使用。该项目将开发方法，在这种方法中，用户的现有数据构成查询的基础，从大量数据集和属性中检索额外的相关数据。要支持这样的查询，有许多技术障碍需要克服。一个主要的挑战是计算效率：该项目将开发快速计算和搜索数据集关系的新算法。研究人员将以丰富的工具为基础，包括随机草图和散列算法，并提供新的理论分析来理解这些方法。贡献的算法将处理高度结构化的数据(例如，时空数据)以及通用的数字或分类数据。第二个挑战是可用性：该项目将开发新的方法来评估所发现的数据关系的重要性，删除巧合或虚假的关系，以及对数据集进行排序并向最终用户展示。最后，该项目将对支持基于数据集关系的广泛可查找性查询的数据集搜索问题做出形式化的贡献。该奖项反映了NSF的法定使命，通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

There has been an explosion in the volume of data that is being collected and cataloged about the environment, society, and populace. Moreover, with the push towards transparency and open data, scientists, governments, and organizations are increasingly making these data available on the Web. Combined with advances in analytics and machine learning, such growing access to data should in theory allow for progress on many of the world’s most important scientific and societal questions. However, this opportunity is often missed due to a central technical barrier: it is currently nearly impossible for domain experts to weed through the vast amount of publicly-available information to discover datasets that are needed for their specific application. Data repository platforms, such as CKAN and Dataverse, and dataset search engines, such as Google Dataset Search, aim to make it easy to share and find datasets. But these systems only support simple, keyword-based queries and metadata search, which are insufficient for users to properly specify their information needs. The investigators envision a new kind of dataset search engine that unlocks the untapped value in open data by supporting a richer set of findability queries that cater to the needs of analytics tasks, and aid in the construction and refinement of machine learning models. By empowering scientists and practitioners with the ability to discover relevant data, the project has great potential to stimulate data reuse both within and across domains.The project will develop methods where the user’s existing data forms the basis of a query that retrieves additional, related data from a large collection of datasets and attributes. There are many technical hurdles to overcome to support such queries. One primary challenge is computational efficiency: this project will develop novel algorithms for rapidly computing and searching for dataset relationships. The investigators will build on a rich variety of tools, including randomized sketching and hashing algorithms, and contribute new theoretical analyses to understand these methods. The algorithms contributed will address both highly-structured data (e.g., spatio-temporal) as well as generic numerical or categorical data. A second challenge is usability: the project will develop novel methods for assessing the significance of discovered data relationships, for pruning out coincidental or spurious relationships, and for ranking and presenting datasets to the end-user. Finally, the project will contribute a formalism to the dataset search problem that supports a wide range of findability queries based on dataset relationships. Active plans for engagement in STEM related activities for high-school students are detailed.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.