III: Medium: Quantifying the Unknown Unknowns for Data Integration

III：媒介：量化数据集成的未知因素

• 批准号: 2033792
• 负责人: Tim Kraska
• 金额: $ 33.45万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-04-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2033792&HistoricalAwards=false
• 关键词: III; Medium; Quantifying; Unknown; Unknowns

As the amount and variety of data available online explodes, it is common practice for data scientists to acquire and integrate disparate data sources to achieve higher quality results. But even with a perfectly cleaned and merged data set, two fundamental questions remain: (1) is the integrated data set complete and (2) what is the impact of any unknown (i.e., unobserved) data on query results? In this work, this project will develop and analyze techniques to estimate the impact of the unknown data (a.k.a., unknown unknowns) for analytical queries. This will help to better understand answers in the presence of incomplete information across fields ranging from business and the military to medical applications.This project will develop and exploit the following paradoxical statistical phenomenon: the ability to see certain data items more than once (across multiple data sets) enables one to estimate parameters of data items that have never been seen at all. This project will therefore develop new statistical techniques which take advantage of overlapping datasets, and software backed by both theory and experiments. This will enable users with overlapping incomplete data sets to actively "see the unseen," and in many cases perform as though they had access to missing information not represented in any of their data sources. The project will also focus on data validation, and how to use multiple unreliable data sources to correct each other. Further, as the proposed analysis is nuanced and novel, the project will also explore how to best convey valuable insights to the user, via interactive visualizations of the predictions. For further information see the project web site at: http://unknown-unknowns.cs.brown.edu

随着可在线可用数据的数量和多种数据爆炸，数据科学家通常是获取和整合不同数据源以获得更高质量结果的常见实践。但是，即使有一个完美清洁和合并的数据集，仍然存在两个基本问题：（1）集成数据集完成，（2）任何未知（即未观察到的）数据对查询结果有什么影响？在这项工作中，该项目将开发和分析技术，以估计未知数据（又称未知未知数）对分析查询的影响。这将有助于更好地理解从业务，军事到医疗应用的领域不完整的信息的存在。该项目将开发和利用以下矛盾的统计统计现象：能够多次查看某些数据项（跨多个数据集）的能力，使一个人可以估算一个从未见过的数据项的一个。因此，该项目将开发新的统计技术，这些技术利用重叠的数据集，并以理论和实验为支持的软件。这将使用户可以重叠不完整的数据集积极地“看到看不见”，并且在许多情况下，他们的执行就好像他们可以访问其任何数据源中未表示的丢失信息一样。 该项目还将集中于数据验证，以及如何使用多个不可靠的数据源来相互纠正。 此外，由于提出的分析是细微的和新颖的，该项目还将通过预测的交互式可视化来探索如何最好地向用户传达有价值的见解。有关更多信息