BIGDATA: F: DKM: Addressing the two V's of Veracity and Variety in Big Data

质量有问题的数据对组织造成了严重的经济和社会负面影响，导致成本超支、收入损失和效率下降。在处理各种各样的数据时，关于数据可靠性、可信度和来源的问题变得更加令人望而生畏，特别是那些不是由组织直接收集的数据，而是来自社交媒体、数据中介和众包等第三方来源的数据。为了解决这些问题，本项目旨在开发一个数据评估引擎(DVE)，以解决数据可靠性、可信度和来源的关键问题，并提供数据获取过程中的问责制和高质量流程。DVE利用和创新估计理论、数据融合和机器学习技术，填补了数据责任和质量方面的关键空白，从而在应对从商业到环境、健康到国家安全的几乎每个应用领域都存在的普遍数据质量问题方面迈出了变革性的一步。DVE将集成到Hadoop生态系统中，与数据源、应用程序或分析无关，并作为托管解决方案提供给社区。用户将通过提供解决问题所需的数据源和相关数据与DVE进行交互。该项目中的DVE将以基本上独立于应用程序的方式开发。开发该引擎的关键挑战包括：(I)如何生成数据质量指示标签，以基于可靠性、可信度、不确定性和置信度等各种因素对数据源和数据内容进行评分？(2)如何整合来自不同来源的具有不同标签分数的数据？(Iii)如何在广泛的应用程序中稳健地评估建议的引擎，这些应用程序充当各种真实世界场景的代理？该研究计划旨在通过一个强有力的评估计划协同应对上述挑战。鉴于所提出的方法、模型和系统的普遍性，该项目将潜在地影响科学、工程和社会科学的各种应用，并具有广泛的环境、经济和健康效益。私人投资机构将发布开放源码软件和适用数据。PIS还将为广泛的用户和参与者提供托管的DVE平台。该项目还在课程和研究经验方面为学生提供了更多接触大数据分析、云计算、数据融合和数据挖掘领域的机会。

Data of questionable quality have led to significantly negative economic and social impacts on organizations, leading to overrun in costs, lost revenue, and decreased efficiencies. The issues on data reliability, credibility, and provenance have become even more daunting when dealing with the variety of data, especially data that are not directly collected by an organization, but from the third-party sources such as social media, data brokers, and crowdsourcing. To address such issues, this project aims to develop a Data Valuation Engine (DVE) that solves the critical problem of data reliability, credibility and provenance, and provides accountability and quality processes right from data acquisition. The DVE leverages and innovates techniques in estimation theory, data fusion and machine learning to fill a critical gap in data accountability and quality, thereby providing a transformative step in countering the ubiquitous data quality issues found in almost every application domain from business to environment to health to national security. The DVE will be integrated in the Hadoop ecosystem and will be agnostic to the data source, application or analytics, and provided as a hosted solution to the community. The user will interact with DVE by providing the data sources and relevant data necessary to solve a problem. The DVE in this project will be developed in a largely application-independent manner. The key challenges to develop this engine include: (i) How to generate the data quality indication labels to score data sources and the content of data based on various factors such as reliability, credibility, uncertainty and confidence? (ii) How to integrate data from various sources with different labeled scores? (iii) How to robustly evaluate the proposed engine in a broad spectrum of applications that serve as a proxy of a variety of real-world scenarios? The research plan has been designed to synergistically address the above challenges with a robust evaluation plan. Given the generality of the proposed methods, models and system, the project will potentially impact variety of applications of science, engineering, and social science and have broad environmental, economic, and health benefits. The PIs will release open source software and applicable data. The PIs will also provide a hosted DVE platform for a broad user and participant base. This project is also providing students with greater exposure to the areas of big data analytics, cloud computing, data fusion and data mining, both in courses and research experiences.