Looking Back to Look Forward: Explaining and Exploring Changes in Data

数据库正在以前所未有的速度收集数据，存储金融和电子商务交易、环境传感器读数和社交网络互动等信息。金融、零售、医疗保健和环境监测中的应用程序在可捕获用户和环境变化的高度动态数据环境中运行。数据不是静态的，它会根据引起数据和元数据(模式)变化的活动自然演变。尽管现有的数据库系统是为数据的存储、管理和高效查询而设计的，但管理和理解数据更改的原因的功能仅限于数据库事务日志、对特定值更改作出反应的触发器以及为查询优化而重新生成的统计信息。 通过了解数据变化的根本原因，数据分析师能够发现变化模式，识别触发大规模变化的事件，并解释数据的谱系和模式演变。这为分析师提供了重要信息，以帮助检测异常和恶意活动、评估对数据质量的信任，或识别趋势和相关性以预测未来的变化。不幸的是，用户通常不知道他们寻求的更改的类型、异常的标准，并且没有有限的工具来了解数据更改如何、何时以及为什么发生。 拟议的研究计划旨在应对上述挑战，并在三个方面产生重大影响。首先，它将在开发变更管理、预测和从异常和不公平更新中清理数据的模型方面推进最先进的技术。这使变化分析能够提高资源分配、库存计划和系统监控应用程序的预测准确性；提高加拿大公司的效率和全球竞争力。其次，它将开发技术来识别变化、其起源和传播之间的空间和时间相互作用。第三，它将开发一个系统，使用户能够衡量和了解其系统变化的后果。该研究计划的长期目标是共同开发数据变化的基础和计算模型。短期目标是设计一个变化探索和解释系统，以提高数据质量。由于几乎一半的新记录包含至少一个关键错误，迫切需要了解数据更改及其对数据质量的影响。我们的系统将帮助用户和企业了解数据是如何演变的，以更快的速度从他们的数据中获得见解，并具有更高的准确性和及时性。

Databases are collecting data at an unprecedented rate, storing information such as financial and e-commerce transactions, environmental sensor readings, and social network interactions. Applications in finance, retail, healthcare, and environmental monitoring operate in highly dynamic data environments that capture user and environmental changes. Data is not static, and naturally evolve according to such activities that elicit change in the data and meta-data (schema). Although existing database systems are designed for storage, management, and efficient querying of data, features to manage and understand the cause of changes to the data are limited to database transaction logs, triggers that react to specific value changes, and statistics that are re-generated for query optimization. By understanding the underlying cause of changes in the data, data analysts are able to discover change patterns, identify events that trigger large-scale changes, and explain the lineage of data and schema evolution. This provides analysts with vital information to help detect abnormal and malicious activity, assess trust for data quality, or to identify trends and correlations to predict future changes. Unfortunately, users are often unaware of the types of changes they are seeking, the criteria for abnormality, and have limited tools to understand how, when and why data changes occur. The proposed research program aims to address the above challenges, and make significant impact along three lines. First, it will advance the state-of-the-art in developing models for change management, prediction, and data cleaning from anomalous and unfair updates. This enables change analysis that increase the predictive accuracy for resource allocation, inventory planning, and system monitoring applications; improving the efficiency and increasing the global competitiveness of Canadian firms. Second, it will develop techniques to identify spatial and temporal interactions between changes, their origins, and their propagation. Third, it will develop a system to enable users to measure and understand the consequential effect of changes in their system. The long-term goal of the research program is to co-develop the foundations and the computational models for data change. The short-term goal is to design a change exploration and explanation system to improve data quality. With almost half of new records containing at least one critical error, understanding data change and its impact on data quality are urgently needed. Our system will help users and businesses understand how data evolves, to derive insights from their data faster, and with greater accuracy and timeliness.