Big data profiling: collecting data about data to support efficient and effective analytics

大数据正在改变人们和企业的决策方式。然而，在投入时间和其他资源来分析海量的可用数据之前，关键是要问这样的问题：“我们是否有适合手头任务的正确数据？”、“我们是否需要在数据适合分析之前清理它们？”或“数据中是否有结构可以帮助我们进行有效和高效的分析？”这些相关的问题可以通过数据概况来回答：收集元数据的活动，即关于数据的数据。例如，在给定电子表格形式的数据集的情况下，有用的元数据可以包括诸如行数、不同值的数目和频繁出现值的标识之类的定量信息，以及诸如列之间的相关性或相关性之类的结构信息。*人们可以通过观察一个小的数据集来分析它，但大数据显然需要自动化技术：当数据量持续增长时，人类的认知处理能力是固定的。拟议的研究计划将为数据分析开发新的方法、算法和软件工具，重点关注大数据的三个“V”字S带来的技术挑战：数量(社交媒体、物联网等产生的数据量不断增长)、速度(数据生成的高速，例如传感器读数或推特消息)和多样性(商业数据、数字数据、图表数据，如社交媒体上的朋友/关注者关系等)。这项研究将在戈拉布博士帮助个人和企业从大数据中获得更多价值的长期研究议程中发挥重要作用。我们迫切需要数据分析工具，让越来越多对将大数据纳入决策过程的专家和非专家更容易获得数据分析。这些工具将帮助加拿大政府、公用事业公司、汽车公司、医疗保健公司和银行更有效和高效地使用大数据。预期的可交付成果也将引起加拿大世界知名数据库公司的兴趣，如IBM多伦多和SAP滑铁卢：提高数据分析性能的关键是利用数据中的结构关系。此外，拟议中的研究将由研究生领导，他们将获得数据科学和大数据工程方面的热门技能，这将有助于他们在加拿大日益由数据驱动的经济中发挥领导作用。

Big data are changing the way people and businesses make decisions. However, before investing time and other resources to analyze the vast amounts of available data, it is critical to ask questions such as "Do we have the right data for the task at hand?", "Do we need to clean the data before they are suitable for analysis?", or "Is there structure in the data that can help us do effective and efficient analytics?". These pertinent questions can be answered through data profiling: the activity of collecting metadata, i.e., data about data. Given a dataset, say, in the form of a spreadsheet, useful metadata may include quantitative information such as the number of rows, the number of distinct values and the identities of frequently occurring values, and structural information such as correlations or dependencies among columns. ******One could profile a small dataset just by looking at it, but automated techniques are clearly needed for big data: while the amount of data keeps growing, human cognitive processing capacity is fixed. The proposed research program will develop new methods, algorithms and software tools for data profiling, focusing on the technical challenges arising from the three “V”s of big data: Volume (the growing amount of data generated by social media, the internet-of-things, etc.), Velocity (the high speed with which data are generated, e.g., sensor readings or twitter messages) and Variety (business data, numeric data, graph data such as friend/follower relationships in social media, etc.). This research will play a major role in Dr. Golab's long-term research agenda to help individuals and businesses get more value out of big data.******Data profiling tools are urgently needed to make data analytics more accessible to the increasing number of experts and non-experts interested in incorporating big data into their decision-making processes. Such tools will help Canadian governments, utilities, automotive companies, healthcare companies and banks to use big data more effectively and efficiently. The anticipated deliverables will also be of interest Canada's world-renowned database companies such as IBM Toronto and SAP Waterloo: a key to improving the performance of data analytics is to exploit structural relationships in the data. Furthermore, the proposed research will be led by graduate students who will acquire sought-after skills in data science and big data engineering, which will help them to take leadership roles in Canada's increasingly data-driven economy.