Query-driven Data Acquisition from Web-based Data Sources

企业和政府机构等各种实体的运作都依赖于获取高质量的数据。这些实体越来越依赖外部来源获取其运营数据：关键数据通过Web服务动态获取、从网页中提取或从第三方购买。这些来源在其完整性、准确性和可用性方面可能存在根本性差异。应用程序不可能在查询之前索引和探索来自每个源的数据：源太多，访问它们的成本太高，并且其中的数据可能会不断刷新。在这种情况下如何进行数据采集？在这个项目中，我们将开发算法，用于回答查询中存在大量的基于Web的数据源，源，可能会重叠，在他们的目录，但有不同的访问限制和成本。我们的方法将利用关于应用程序正在查询的数据的模式信息：数据格式、完整性约束和任何可能可用的成本先验知识。该项目的核心将是通过交互式地探索源来回答查询的算法，在这个过程中动态地修剪掉不相关的或耗尽的源。

The functioning of entities as diverse as enterprises and government agencies depends onobtaining high-quality data.Increasingly these entities depend on external sourcesfor their operational data: critical datais obtained dynamically via web services, is extracted from web pages,or is purchased from third parties. These sources can differ radicallyin their completeness, accuracy, and availability. It is not possible for applications to indexand explore data from each source in advance of querying:there are too many sources, they are too costly to access, and the data in themmay be refreshed constantly. How should data acquisition proceed in such situations?In this project we will develop algorithms for answering queries in the presence of large numbers ofweb-based data sources, sources that may overlap substantially in their datasetsbut have different access restrictions and costs. Our approach will make use of schema information about thedata an application is querying: data format, integrity constraints, and any prior knowledge of costs that maybe available. The core of the project will be algorithms for answering a query by interactively exploring the sources,dynamically pruning out irrelevant or exhausted sources in the process.