III: Small: Managing Large-scale Uncertain Data Repositories

越来越多的实际应用领域产生的数据本质上是有噪声的、不完整的和概率性的。此类数据的示例包括传感器网络收集的测量数据、社交网络中的观察数据、科学和生物医学数据以及各种在线网络来源收集的数据。数据不确定性可能是由于潜在测量基础设施的基本限制，领域内固有的模糊性，或者它们可能是通常用于从传感器和网络数据中提取级别事件的丰富概率建模的副作用。类似地，当试图集成异构数据源（“数据集成”）或从文本中提取结构化信息（“信息提取”）时，结果充其量是近似的和不确定的。然而，目前缺乏能够对大量不确定数据进行推理的数据管理工具，因此，关于不确定性的信息通常要么被丢弃，要么只是肤浅地进行推理。在这个项目中，我们正在构建一个完整的概率数据管理系统，称为PrDB，它可以管理、存储和处理大规模的不确定数据库。PrDB统一了机器学习文献中开发的“大规模结构化图形模型”（如概率关系模型（PRMs））和数据库文献中研究的“概率查询处理”的思想。PrDB框架基于“共享因素”的概念，它不仅允许我们在不同抽象层次上表达和操纵不确定性，而且还支持捕获不确定数据之间丰富的相关性。PrDB支持一种类似sql的声明性语言，用于指定不确定数据及其之间的相关性。PrDB还支持各种查询的精确和近似求值，包括推理查询、SQL查询和决策支持查询。在这个项目中进行的跨学科研究将使我们能够同时解决概率数据库和机器学习领域的挑战，并允许我们在这些领域之间转移开发的关键技术，从而推进这两个领域的研究。它将使在健康信息学、社会媒体管理、万维网和科学数据库等各种领域开发重要的、高影响力的新型现实世界应用成为可能。PrDB系统源代码和项目期间生成的数据集将使用适当的开源许可证在项目网站上发布：http://www.cs.umd.edu/db/PrDB.html

Increasing numbers of real-world application domains are generating data that isinherently noisy, incomplete, and probabilistic in nature. Examples of suchdata include measurement data collected by sensor networks, observation data inthe context of social networks, scientific and biomedical data, and datacollected by various online cyber-sources. The data uncertainties may be aresult of the fundamental limitations of the underlying measurementinfrastructures, the inherent ambiguity in the domain, or they may be aside-effect of the rich probabilistic modeling typically performed to extracthigh-level events from sensor and cyber data. Similarly, when attempting tointegrate heterogeneous data sources ("data integration") or extractingstructured information from text ("information extraction"), the results areapproximate and uncertain at best. However, there is currently a lack of datamanagement tools that can reason about large volumes of uncertain data, andhence the information about the uncertainty is often either discarded orreasoned about only superficially.In this project, we are building a complete probabilistic data managementsystem, called PrDB, that can manage, store, and process large-scalerepositories of uncertain data. PrDB unifies ideas from "large-scale structuredgraphical models" like probabilistic relational models (PRMs), developed in themachine learning literature, and "probabilistic query processing", studied inthe database literature. PrDB framework is based on the notion of "sharedfactors", which not only allows us to express and manipulate uncertainties atvarious levels of abstractions, but also supports capturing rich correlationsamong the uncertain data. PrDB supports a declarative SQL-like language forspecifying uncertain data and the correlations among them. PrDB also supportsexact and approximate evaluation of a wide range of queries including inferencequeries, SQL queries, and decision-support queries.The cross-disciplinary research undertaken during this project will enable us tosimultaneously address the challenges in the areas of probabilistic databasesand machine learning, and allow us to transfer the key technologies developedbetween those areas, thus advancing the research in both areas. It will enablethe development of a significant and high-impact new class of real-worldapplications, in a variety of domains including health informatics, social mediamanagement, World Wide Web, and scientific databases. The PrDB system sourcecode, and the datasets generated during the project, will be released using anappropriate open source license, at the project web site:http://www.cs.umd.edu/db/PrDB.html