Looking Back to Look Forward: Explaining and Exploring Changes in Data

点击翻译按钮获取中文摘要

Databases are collecting data at an unprecedented rate, storing information such as financial and e-commerce transactions, environmental sensor readings, and social network interactions.  Applications in finance, retail, healthcare, and environmental monitoring operate in highly dynamic data environments that capture user and environmental changes. Data is not static, and naturally evolve according to such activities that elicit change in the data and meta-data (schema). Although existing database systems are designed for storage, management, and efficient querying of data, features to manage and understand the cause of changes to the data are limited to database transaction logs, triggers that react to specific value changes, and statistics that are re-generated for query optimization.   By understanding the underlying cause of changes in the data, data analysts are able to discover change patterns, identify events that trigger large-scale changes, and explain the lineage of data and schema evolution. This provides analysts with vital information to help detect abnormal and malicious activity, assess trust for data quality, or to identify trends and correlations to predict future changes. Unfortunately, users are often unaware of the types of changes they are seeking, the criteria for abnormality, and have limited tools to understand how, when and why data changes occur.  The proposed research program aims to address the above challenges, and make significant impact along three lines. First, it will advance the state-of-the-art in developing models for change management, prediction, and data cleaning from anomalous and unfair updates.  This enables change analysis that increase the predictive accuracy for resource allocation, inventory planning, and system monitoring applications; improving the efficiency and increasing the global competitiveness of Canadian firms.  Second, it will develop techniques to identify spatial and temporal interactions between changes, their origins, and their propagation. Third, it will develop a system to enable users to measure and understand the consequential effect of changes in their system.  The long-term goal of the research program is to co-develop the foundations and the computational models for data change. The short-term goal is to design a change exploration and explanation system to improve data quality.  With almost half of new records containing at least one critical error, understanding data change and its impact on data quality are urgently needed.  Our system will help users and businesses understand how data evolves, to derive insights from their data faster, and with greater accuracy and timeliness.