Integrating third-party and open data with internal corporate databases

许多公司希望更好地了解客户的需求，以努力帮助他们实现目标并保持积极的相互关系。在典型的设置中，有关客户（如姓名和地址）及其与公司的交易的信息存储在由公司管理的专用数据库中。然而，额外的信息通常可以通过其他来源获得，公司可以访问其中的一些资源（例如环境、财富、保险等），也可以访问公开的数据。利用第三方和公共数据（统称为外部数据）有望带来更大的价值，帮助公司更接近其目标，但也会带来挑战。首先，就模式、表和列的命名以及完整性约束而言，不期望外部数据遵循管理内部数据的同一组规则。关于实体的信息可以分布在多个（有时是不一致的）来源上。其次，一个实体的识别细节在不同的来源之间可能不同（例如，由于拼写和其他变化），并且可能无法确定地连接来自这些来源的数据。第三，数据可能具有一些空间或时间属性。例如，内容可能与地理位置相关联。理解这些属性对于解决一些歧义可能很重要。

Many companies want to better understand the needs of their customers, in an effort to help them reach their goals and to maintain a positive mutual relationship. In a typical setting, information about customers (such as name and address) and their transactions with a company is stored in a dedicated database managed by the company. However, additional information is often available through other sources and the company may have access to some of these resources (e.g. Environics, Wealth, Insurance, etc.) and also to the data that is available publicly. Utilizing third party and public data (collectively referred to as external data) is expected to bring greater value, helping the company move closer to some of its objectives, but also introduces challenges. First, external data is not expected to follow the same set of rules governing the internal data, in terms of the schema, naming of the tables and columns and integrity constraints. Information about an entity can be spread over multiple (and sometimes inconsistent) sources. Second, the identifying detail of an entity can differ between sources (e.g. due to spelling and other variations) and it may not be possible to join the data from such sources with certainty. Third, data may have some spatial or temporal attributes. For example, the content may be associated to a geographical location. Understanding these attributes may be important in resolving some of the ambiguities. The primary objective of this research is to study the challenges in integrating third-party and open data with data residing inside an organization. We will investigate robust system architectures and efficient and scalable algorithms that support integration at different stages including data cleaning, schema mapping, and query processing. This project will be developed in partnership with Servus Credit Union, an Alberta-based company that deals with the aforementioned integration and cleaning challenges to support down-the-stream data analytics applications and services the company runs.