BIGDATA: F: DKM: Addressing the two V's of Veracity and Variety in Big Data

质量有问题的数据对各组织造成了严重的负面经济和社会影响，导致成本超支、收入损失和效率下降。在处理各种数据时，数据可靠性，可信度和来源的问题变得更加艰巨，特别是不是由组织直接收集的数据，而是来自第三方来源，如社交媒体，数据经纪人和众包。为了解决这些问题，该项目旨在开发一个数据评估引擎（DVE），解决数据可靠性，可信度和出处的关键问题，并从数据采集开始提供问责制和质量流程。DVE利用并创新了估计理论、数据融合和机器学习方面的技术，以填补数据问责制和质量方面的关键空白，从而为应对从商业到环境、健康到国家安全的几乎每个应用领域中普遍存在的数据质量问题迈出了变革性的一步。DVE将被集成到Hadoop生态系统中，并将与数据源、应用程序或分析无关，并作为托管解决方案提供给社区。 用户将通过提供解决问题所需的数据源和相关数据与DVE进行交互。在这个项目中的DVE将在很大程度上独立于应用程序的方式开发。开发该引擎的关键挑战包括：（i）如何生成数据质量指示标签，以基于可靠性、可信度、不确定性和置信度等各种因素对数据源和数据内容进行评分？(ii)如何整合来自不同来源的具有不同标记分数的数据？(iii)如何在广泛的应用程序中稳健地评估所提出的引擎，这些应用程序可作为各种真实场景的代理？该研究计划旨在通过一个强有力的评价计划协同应对上述挑战。鉴于所提出的方法、模型和系统的通用性，该项目将潜在地影响科学、工程和社会科学的各种应用，并具有广泛的环境、经济和健康效益。PI将发布开源软件和适用数据。PI还将为广泛的用户和参与者基础提供托管DVE平台。该项目还为学生提供了更多的接触大数据分析，云计算，数据融合和数据挖掘领域，无论是在课程和研究经验。

Data of questionable quality have led to significantly negative economic and social impacts on organizations, leading to overrun in costs, lost revenue, and decreased efficiencies. The issues on data reliability, credibility, and provenance have become even more daunting when dealing with the variety of data, especially data that are not directly collected by an organization, but from the third-party sources such as social media, data brokers, and crowdsourcing. To address such issues, this project aims to develop a Data Valuation Engine (DVE) that solves the critical problem of data reliability, credibility and provenance, and provides accountability and quality processes right from data acquisition. The DVE leverages and innovates techniques in estimation theory, data fusion and machine learning to fill a critical gap in data accountability and quality, thereby providing a transformative step in countering the ubiquitous data quality issues found in almost every application domain from business to environment to health to national security. The DVE will be integrated in the Hadoop ecosystem and will be agnostic to the data source, application or analytics, and provided as a hosted solution to the community. The user will interact with DVE by providing the data sources and relevant data necessary to solve a problem. The DVE in this project will be developed in a largely application-independent manner. The key challenges to develop this engine include: (i) How to generate the data quality indication labels to score data sources and the content of data based on various factors such as reliability, credibility, uncertainty and confidence? (ii) How to integrate data from various sources with different labeled scores? (iii) How to robustly evaluate the proposed engine in a broad spectrum of applications that serve as a proxy of a variety of real-world scenarios? The research plan has been designed to synergistically address the above challenges with a robust evaluation plan. Given the generality of the proposed methods, models and system, the project will potentially impact variety of applications of science, engineering, and social science and have broad environmental, economic, and health benefits. The PIs will release open source software and applicable data. The PIs will also provide a hosted DVE platform for a broad user and participant base. This project is also providing students with greater exposure to the areas of big data analytics, cloud computing, data fusion and data mining, both in courses and research experiences.