III: Small: Managing Large-scale Uncertain Data Repositories

越来越多的真实应用程序领域正在生成本质上具有噪声、不完整和概率性质的数据。这类数据的例子包括传感器网络收集的测量数据、社交网络背景下的观测数据、科学和生物医学数据以及各种在线网络来源收集的数据。数据不确定性可能是基础测量基础设施的基本限制、领域中固有的模棱两可的结果，也可能是通常为从传感器和网络数据中提取高级别事件而执行的丰富的概率建模的副效应。同样，当尝试集成不同的数据源(“数据集成”)或从文本中提取结构化信息(“信息提取”)时，结果充其量是近似的和不确定的。然而，目前缺乏能够对大量不确定数据进行推理的数据管理工具，因此关于不确定性的信息往往要么被丢弃，要么只是表面上的推理。在本项目中，我们正在构建一个完整的概率数据管理系统PrDB，它可以管理、存储和处理大规模的不确定数据。PrDB统一了“大规模结构化图形模型”的思想，如在机器学习文献中开发的概率关系模型(PRM)，以及在数据库文献中研究的“概率查询处理”。PrDB框架基于“共享因子”的概念，不仅允许我们表达和操纵不同抽象层次的不确定性，而且支持捕获不确定数据之间的丰富关联。PrDB支持一种声明性的类似SQL的语言，用于指定不确定数据及其之间的相关性。PrDB还支持对各种查询的性别和近似评估，包括推理查询、SQL查询和决策支持查询。在此项目期间进行的跨学科研究将使我们能够同时应对概率数据库和机器学习领域的挑战，并允许我们在这些领域之间转移开发的关键技术，从而推动这两个领域的研究。它将在包括健康信息学、社会媒体管理、万维网和科学数据库在内的多个领域开发一类重要的、高影响力的现实世界应用程序。PrDB系统源代码和项目期间生成的数据集将使用适当的开放源码许可证在项目网站上发布：http://www.cs.umd.edu/db/PrDB.html

Increasing numbers of real-world application domains are generating data that isinherently noisy, incomplete, and probabilistic in nature. Examples of suchdata include measurement data collected by sensor networks, observation data inthe context of social networks, scientific and biomedical data, and datacollected by various online cyber-sources. The data uncertainties may be aresult of the fundamental limitations of the underlying measurementinfrastructures, the inherent ambiguity in the domain, or they may be aside-effect of the rich probabilistic modeling typically performed to extracthigh-level events from sensor and cyber data. Similarly, when attempting tointegrate heterogeneous data sources ("data integration") or extractingstructured information from text ("information extraction"), the results areapproximate and uncertain at best. However, there is currently a lack of datamanagement tools that can reason about large volumes of uncertain data, andhence the information about the uncertainty is often either discarded orreasoned about only superficially.In this project, we are building a complete probabilistic data managementsystem, called PrDB, that can manage, store, and process large-scalerepositories of uncertain data. PrDB unifies ideas from "large-scale structuredgraphical models" like probabilistic relational models (PRMs), developed in themachine learning literature, and "probabilistic query processing", studied inthe database literature. PrDB framework is based on the notion of "sharedfactors", which not only allows us to express and manipulate uncertainties atvarious levels of abstractions, but also supports capturing rich correlationsamong the uncertain data. PrDB supports a declarative SQL-like language forspecifying uncertain data and the correlations among them. PrDB also supportsexact and approximate evaluation of a wide range of queries including inferencequeries, SQL queries, and decision-support queries.The cross-disciplinary research undertaken during this project will enable us tosimultaneously address the challenges in the areas of probabilistic databasesand machine learning, and allow us to transfer the key technologies developedbetween those areas, thus advancing the research in both areas. It will enablethe development of a significant and high-impact new class of real-worldapplications, in a variety of domains including health informatics, social mediamanagement, World Wide Web, and scientific databases. The PrDB system sourcecode, and the datasets generated during the project, will be released using anappropriate open source license, at the project web site:http://www.cs.umd.edu/db/PrDB.html