Big data profiling: collecting data about data to support efficient and effective analytics

点击翻译按钮获取中文摘要

Big data are changing the way people and businesses make decisions. However, before investing time and other resources to analyze the vast amounts of available data, it is critical to ask questions such as "Do we have the right data for the task at hand?", "Do we need to clean the data before they are suitable for analysis?", or "Is there structure in the data that can help us do effective and efficient analytics?". These pertinent questions can be answered through data profiling: the activity of collecting metadata, i.e., data about data. Given a dataset, say, in the form of a spreadsheet, useful metadata may include quantitative information such as the number of rows, the number of distinct values and the identities of frequently occurring values, and structural information such as correlations or dependencies among columns.