Integrating third-party and open data with internal corporate databases

许多公司希望更好地了解客户的需求，以帮助他们实现目标并保持积极的相互关系。在典型的设置中，关于客户的信息（例如姓名和地址）以及他们与公司的交易存储在由公司管理的专用数据库中。但是，其他信息通常可以通过其他来源获得，公司可能可以访问其中一些资源（例如环境、财富、保险等）。以及公开可用的数据。利用第三方和公共数据（统称为外部数据）有望带来更大的价值，帮助公司更接近其一些目标，但也带来了挑战。首先，外部数据在模式、表和列的命名以及完整性约束方面不应遵循管理内部数据的同一组规则。关于一个实体的信息可以分散在多个（有时不一致）来源。第二，实体的识别细节在不同来源之间可能不同（例如，由于拼写和其他变化），并且可能无法确定地连接来自这些来源的数据。第三，数据可能具有一些空间或时间属性。例如，内容可以与地理位置相关联。理解这些属性对于解决某些模糊性可能很重要。 本研究的主要目标是研究将第三方和开放数据与组织内部的数据集成的挑战。我们将研究健壮的系统架构和高效的可扩展算法，这些算法支持不同阶段的集成，包括数据清理、模式映射和查询处理。该项目将与Servus Credit Union合作开发，Servus Credit Union是一家位于艾伯塔省的公司，负责处理上述集成和清理挑战，以支持公司运行的下游数据分析应用程序和服务。

Many companies want to better understand the needs of their customers, in an effort to help them reach their goals and to maintain a positive mutual relationship. In a typical setting, information about customers (such as name and address) and their transactions with a company is stored in a dedicated database managed by the company. However, additional information is often available through other sources and the company may have access to some of these resources (e.g. Environics, Wealth, Insurance, etc.) and also to the data that is available publicly. Utilizing third party and public data (collectively referred to as external data) is expected to bring greater value, helping the company move closer to some of its objectives, but also introduces challenges. First, external data is not expected to follow the same set of rules governing the internal data, in terms of the schema, naming of the tables and columns and integrity constraints. Information about an entity can be spread over multiple (and sometimes inconsistent) sources. Second, the identifying detail of an entity can differ between sources (e.g. due to spelling and other variations) and it may not be possible to join the data from such sources with certainty. Third, data may have some spatial or temporal attributes. For example, the content may be associated to a geographical location. Understanding these attributes may be important in resolving some of the ambiguities. The primary objective of this research is to study the challenges in integrating third-party and open data with data residing inside an organization. We will investigate robust system architectures and efficient and scalable algorithms that support integration at different stages including data cleaning, schema mapping, and query processing. This project will be developed in partnership with Servus Credit Union, an Alberta-based company that deals with the aforementioned integration and cleaning challenges to support down-the-stream data analytics applications and services the company runs.