Data Integration: from NoSQL to Main Memory Data Sources

点击翻译按钮获取中文摘要

Consider that a modern enterprise will have access to a large number of structured data sources that are managed by internal operational information systems or that can be accessed through external sites on the web. The sources will provide interfaces that can range from low-level file systems storing such artifacts as CSV files, to intermediate-level key-value systems managing very large volumes of operational data, through to high-level relational database systems providing access to structured data via SQL. Also consider that the enterprise will have an abundance of circumstances where code must to be written by developers to access many of these data sources at runtime in order to satisfy emerging integration requirements for its existing information systems, or to satisfy the requirements of new information systems currently under development. Such circumstances are instances of the general problem of data integration, the focus of this research agenda. Data integration is the problem of finding and executing query plans that manifest efficient ways to interact with data sources so as to obtain knowledge of their structured data, and to combine the results of this interaction in a way that, as efficiently as possible, yields the answers to user queries coded by developers. The problem becomes particularly challenging when user queries are expressed in a high-level non-procedural language, such as SQL, over a logical design. Such a design, sometimes called a mediated schema or ontology, is a problem or domain specific appreciation of the data entirely devoid of any knowledge of the data sources themselves. The increase in developer productivity enabled by data integration technology that provides such a developer interface is now indisputable; witness that relational technology continues to be a growing multi-billion dollar industry. There are three subproblems in data integration: (1) the selection of suitable languages for capturing so-called physical designs (physical designs augment logical designs with the source schemas of data residing at various data sources, with mapping rules relating the logical design to source schemas, and with knowledge of the capabilities of the data sources themselves), (2) the selection of suitable languages for expressing user queries and query plans, and (3) the problem of finding query plans given physical designs and user queries. This project is a continuation of my ongoing work on each of these subproblems, and therefore also adopts the hypothesis that first order logic (FOL) and so-called closed world semantics assumed by the SQL standard are an appropriate starting point or foundation for an underlying theory and technology of data integration. Indeed, FOL lies at the heart of relational database systems and the relational model of data, and has had a profound influence, via relational algebra, on the development of the SQL query language.