Continuous Data Curation

点击翻译按钮获取中文摘要

With interest in analysis of data at an all-time high data curation has become a critical challenge. Data curation includes profiling, cleaning and managing data in preparation for data analysis. With the ascendance of big data, many information technology leaders are neglecting the price of admission to the big data world of data preparation. Big data initiatives are likely to take longer, cost more, and deliver fewer benefits without curated data. The ability to store data is not a problem anymore, according to a survey of senior executives conducted by the Economist Intelligence Unit in 2017 less than 20% indicated data storage as a problem, however, more than 50% rated other data management tasks, such as reconciliation, integration and cleaning as problematic. Forbes in 2017 assessed that data curation accounts for around 80% of the work of data scientists. Data curation is so problematic and time consuming because of the lack of tools, scientific frameworks, and theoretical foundations to support principled data preparation. However, without principled data management and preparation, new data analytic insights cannot be trusted. The proposed research will develop novel methods and software tools for large-scale data curation, focusing on the technical challenges arising from the four Vs of big data: Volume, Velocity and Variety and Veracity. First, given the dynamic nature of big data, we will develop new continuous data profiling methods. One could profile a small dataset just by looking at it, however, automated (and incremental) techniques are clearly needed for big data. While the amount of available and potentially useful data keeps growing, human cognitive processing capacity is fixed. Second, recognizing the massive heterogeneity of big data and that automation will rarely provide 100% accuracy, we will investigate the use of provenance and domains ontologies over streaming data to enhance rule-based data cleaning. Third, we will build efficient problem determination and adaptive database management system tuning tools through distributed computing and machine learning. These new data curation techniques are beneficial to governmental and business organizations. Organizations can significantly benefit by making analytical decisions over high quality data. Our solutions can be utilized in healthcare (Toronto General Hospital), telecommunication (ATs position as one of the leaders in Information and Communication Technologies (ICT). Furthermore, the proposed research will create a unique training environment, in which students will acquire sought-after experience in data science, one of the fastest-growing disciplines within ICT worldwide.