Big data profiling: collecting data about data to support efficient and effective analytics

大数据正在改变人们和企业做出决策的方式。然而，在投入时间和其他资源来分析大量可用数据之前，关键是要问这样的问题：“我们手头的任务是否有正确的数据？“，“我们是否需要在数据适合分析之前对其进行清理？或者“数据中是否有结构可以帮助我们进行有效和高效的分析？".这些相关的问题可以通过数据分析来回答：收集元数据的活动，即，关于数据的数据给定一个数据集，比如说，以电子表格的形式，有用的元数据可以包括定量信息，如行数，不同值的数量和频繁出现的值的身份，以及结构信息，如列之间的相关性或依赖性。人们可以通过观察来分析一个小数据集，但大数据显然需要自动化技术：虽然数据量不断增长，但人类的认知处理能力是固定的。拟议的研究计划将开发用于数据分析的新方法，算法和软件工具，重点关注大数据的三个“V”所带来的技术挑战：卷（社交媒体，物联网等产生的数据量不断增加），速度（生成数据的高速度，例如，传感器读数或Twitter消息）和多样性（商业数据、数字数据、图形数据，诸如社交媒体中的朋友/追随者关系等）。这项研究将在Golab博士的长期研究议程中发挥重要作用，以帮助个人和企业从大数据中获得更多价值。迫切需要数据分析工具，以使越来越多的专家和非专家更容易将大数据纳入决策过程。这些工具将帮助加拿大政府、公用事业、汽车公司、医疗保健公司和银行更有效地利用大数据。预期的交付成果也将引起加拿大世界知名的数据库公司的兴趣，如IBM多伦多和SAP滑铁卢：提高数据分析性能的关键是利用数据中的结构关系。此外，拟议的研究将由研究生领导，他们将获得数据科学和大数据工程方面的抢手技能，这将有助于他们在加拿大日益数据驱动的经济中发挥领导作用。

Big data are changing the way people and businesses make decisions. However, before investing time and other resources to analyze the vast amounts of available data, it is critical to ask questions such as "Do we have the right data for the task at hand?", "Do we need to clean the data before they are suitable for analysis?", or "Is there structure in the data that can help us do effective and efficient analytics?". These pertinent questions can be answered through data profiling: the activity of collecting metadata, i.e., data about data. Given a dataset, say, in the form of a spreadsheet, useful metadata may include quantitative information such as the number of rows, the number of distinct values and the identities of frequently occurring values, and structural information such as correlations or dependencies among columns. One could profile a small dataset just by looking at it, but automated techniques are clearly needed for big data: while the amount of data keeps growing, human cognitive processing capacity is fixed. The proposed research program will develop new methods, algorithms and software tools for data profiling, focusing on the technical challenges arising from the three “V”s of big data: Volume (the growing amount of data generated by social media, the internet-of-things, etc.), Velocity (the high speed with which data are generated, e.g., sensor readings or twitter messages) and Variety (business data, numeric data, graph data such as friend/follower relationships in social media, etc.). This research will play a major role in Dr. Golab's long-term research agenda to help individuals and businesses get more value out of big data.Data profiling tools are urgently needed to make data analytics more accessible to the increasing number of experts and non-experts interested in incorporating big data into their decision-making processes. Such tools will help Canadian governments, utilities, automotive companies, healthcare companies and banks to use big data more effectively and efficiently. The anticipated deliverables will also be of interest Canada's world-renowned database companies such as IBM Toronto and SAP Waterloo: a key to improving the performance of data analytics is to exploit structural relationships in the data. Furthermore, the proposed research will be led by graduate students who will acquire sought-after skills in data science and big data engineering, which will help them to take leadership roles in Canada's increasingly data-driven economy.