Integrating third-party and open data with internal corporate databases

点击翻译按钮获取中文摘要

Many companies want to better understand the needs of their customers, in an effort to help them reach their goals and to maintain a positive mutual relationship. In a typical setting, information about customers (such as name and address) and their transactions with a company is stored in a dedicated database managed by the company. However, additional information is often available through other sources and the company may have access to some of these resources (e.g. Environics, Wealth, Insurance, etc.) and also to the data that is available publicly. Utilizing third party and public data (collectively referred to as external data) is expected to bring greater value, helping the company move closer to some of its objectives, but also introduces challenges. First, external data is not expected to follow the same set of rules governing the internal data, in terms of the schema, naming of the tables and columns and integrity constraints. Information about an entity can be spread over multiple (and sometimes inconsistent) sources. Second, the identifying detail of an entity can differ between sources (e.g. due to spelling and other variations) and it may not be possible to join the data from such sources with certainty. Third, data may have some spatial or temporal attributes. For example, the content may be associated to a geographical location. Understanding these attributes may be important in resolving some of the ambiguities.The primary objective of this research is to study the challenges in integrating third-party and open data with data residing inside an organization. We will investigate robust system architectures and efficient and scalable algorithms that support integration at different stages including data cleaning, schema mapping, and query processing. This project will be developed in partnership with Servus Credit Union, an Alberta-based company that deals with the aforementioned integration and cleaning challenges to support down-the-stream data analytics applications and services the company runs.