III: Small: Linking and Resolving Entities in Big Data

该项目将探讨在大数据分析管道的背景下清理数据的挑战。传统上，数据清理旨在提高ETL系统中的数据质量，在ETL系统中，企业数据被收集、准备、暂存、转换并加载到数据仓库中，以支持离线数据分析。在大数据时代，这样的后端流程正在迅速让位于交互式探索性数据分析，分析师沉浸在数据中(可能是从不同的数据源收集)，以推动在线(接近)实时决策。现有系统不能适应动态生成的数据(例如，社交媒体流)的数量、速度或可变性，并且离线体系结构不适合分析的在线实时性质。市场上充斥着数据转换技术的创新，例如，TriFacta允许分析师可视化地操作数据以生成复杂的分析转换，而Data Tamer正在探索从不同来源进行可扩展的数据精选。数据质量(以及数据清理技术)仍然是大数据分析的核心。许多流行媒体(以及学术)文章强调，实体链接和解析等挑战是大数据分析最重要、最直接的障碍之一。此项目基于的关键见解是，数据清理以支持对大数据的分析，不仅仅是通过利用更多硬件来扩展已知的方法来处理更大的数据集。虽然纵向扩展很重要，但流、实时和交互环境中的大数据分析需要在执行数据清理的方式上进行范式转变。该项目将显著影响和改变数据清理的现代实践以及将清理整合到大数据分析管道中的方式，并将通过以下方式探索更广泛的影响：(A)与相关行业合作伙伴的技术转让机会，其现有产品可从拟议的研究中受益；以及(B)在目前正在开发的社交媒体分析系统(SODA)的背景下的开源努力，其中将整合拟议的研究算法。本研究将探索两项新的创新，这将有助于推进数据清理，以实现大数据分析。第一个创新探索了一种渐进的实体解析方法，以支持渐进分析。这项研究将探索一种方法，其中进步性贯穿清洁过程的所有阶段，特别是在清洁基于复杂逻辑的情况下，可能需要动态获取额外的上下文信息。第二个创新是针对在静态和流数据之上发布的一次性和连续查询场景的结构化查询(例如，配置单元和SQL)开发的具有分析意识的数据清理。该项目将在更高的概念层面处理这些方法，并在运行在机器集群上的现代高度并行计算平台和框架上实施这些方法。该项目将利用两个具体背景来指导研究探索：(A)支持对融合表等结构化网络数据源进行分析查询；(B)在线分析社交媒体数据。这些应用背景将作为测试和演示研究的工具。计划中的研究、系统开发和教育活动(例如，将与大数据和数据质量相关的项目纳入加州大学洛杉矶分校CS课程的课程改革)将显著增强学生的教育体验，为他们在当今S知识驱动的社会中更美好的未来做好准备。有关该项目的更多信息，请访问http://sherlock.ics.uci.edu.。

This project will explore the challenge of cleaning data in the context of analysis pipelines over big data. Data cleaning has traditionally been designed to improve data quality in ETL systems where enterprise data is collected, prepared, staged, transformed, and loaded into a data warehouse to support offline data analysis. In the era of big data, such back-end processes are quickly giving way to interactive exploratory data analysis where analysts immerse themselves in data (possibly collected from heterogeneous data sources) in order to drive online (near-) real-time decision making. Existing systems do not scale to the volume, velocity, or the variability of the dynamically generated data (e.g., social media streams) and the offline architecture is unsuited for the online real-time nature of analysis. The market is abuzz with innovations in data transformation technologies, e.g., TriFacta allows analysts to visually manipulate data to generate complex analytical transformations and Data Tamer is exploring scalable data curation from diverse sources. Data quality (and hence data cleaning technologies) remain at the core of big-data analytics. Many popular media (as well as academic) articles have highlighted challenges such as entity linking and resolution as among the most important and immediate roadblocks for big data analytics. The key insight on which this project is based is that data cleaning to support analytics over big data is not simply a matter of scaling up known approaches to larger data sets by exploiting more hardware. While scale up is important, big data analytics in streaming, real-time, and interactive settings requires a paradigm shift in how data cleaning is performed. This project will significantly impact and change the modern practices of data cleaning and the way cleaning is integrated in the Big Data analysis pipeline and will explore broader impact through: (a) technology transfer opportunities with a relevant industrial partner whose existing products could benefit from the proposed research; and (b) open source effort in the context of the ongoing social media analytics system (SoDAS), currently under development, in which the proposed research algorithms will be integrated.This research will explore two new innovations that will help advance data cleaning to enable Big Data analysis. The first innovation explores a progressive approach to entity resolution to support progressive analysis. The research will explore an approach where progressiveness is pervasive spanning all the phases of the cleaning process especially in scenarios when cleaning is based on complex logic possibly requiring dynamic acquisition of additional contextual information. The second innovation is the analysis-aware data cleaning that is developed for structured queries (e.g., Hive and SQL) for both one-time and continuous query scenarios that are issued on top of static and streaming data. The project will address these methodologies at the higher conceptual level as well as implement them on modern highly-parallel computing platforms and frameworks that run on a cluster of machines. The project will exploit two concrete contexts to guide the research exploration: (a) supporting analytical queries over structured web data sources such as fusion tables; and (b) online analysis of social media data. These application contexts will serve as vehicles for testing and demonstrating the research. The planned research, system development, and educational activities (e.g., curriculum changes to incorporate projects related to big data and data quality in the CS curriculum at UCI) will significantly enhance the educational experience of students, preparing them for a brighter future in the today?s knowledge driven society. More information about the project can be found at http://sherlock.ics.uci.edu.