CICI: Data Provenance: Provenance-Based Trust Management for Collaborative Data Curation

数据驱动的科学不仅依赖于统计和机器学习，还依赖于人类的专业知识。由于收集数据是为了解决日益具有挑战性的科学和医学问题，因此需要相应地扩大专家人力投入（管理和某些情况下的注释）的数量。这个项目通过开发协作数据管理解决了这一需求：它不依赖于少数专家，而是允许不同专业知识的用户社区进行注释。由于不同用户的注释质量会有所不同，因此开发了新的定量技术来评估每个用户的可信度，基于他们的行为，并将值得信赖的专家与不熟练和恶意的用户区分开来。开发了基于可信度的用户注释组合算法。协作数据管理将大大增加人类注释数据的数量，这反过来又将为生命科学、医学等领域带来更好的大数据分析和检测算法。协作数据管理的核心问题在于用户注释质量的高度可变性，以及用户注释时数据形式的可变性。该提案开发了一些技术，以获取不同用户对不同数据视图所做的注释（例如对信号应用过滤器和转换的脑电图显示），使用来源来推断注释与原始数据的关系，并推断每个用户对该数据的注释的可靠性和可信度。为了实现这一目标，该研究首先定义了数据和来源模型，以捕获随时间和空间变化的数据；基于个人的注释以及这些注释与来自公认专家和更广泛社区的注释的比较，用于计算和动态更新个人的可靠性和可信度的新型可靠性演算算法；以及一种名为PAL的高级语言，使研究人员能够实施和比较多种策略。研究人员将首先开发和验证神经科学和时间序列数据的技术，在900多个用户的公共数据共享门户中（有1500多个EEG和其他需要注释的数据集）。项目团队随后将该技术扩展到其他数据模式，如成像和基因组学

Data-driven science relies not only on statistics and machine learning, but also on human expertise. As data are being collected to tackle increasingly challenging scientific and medical problems, there is need to scale up the amount of expert human input (curation and, in certain cases, annotation) accordingly. This project addresses this need by developing collaborative data curation: instead of relying on a small number of experts, it enables annotations to be made by communities of users of varying expertise. Since the quality of annotations by different users will vary, novel quantitative techniques are developed to assess the trustworthiness of each user, based on their actions, and to distinguish trustworthy experts from unskilled and malicious users. Algorithms are developed to combine users' annotations based on their trustworthiness. Collaborative data curation will greatly increase the amount of human annotated data, which will, in turn, lead to better Big Data analysis and detection algorithms for the life sciences, medicine, and beyond.The central problems of collaborative data curation lie in the high variability in the quality of users' annotations, and variability in the form the data takes when they annotate it. The proposal develops techniques to take annotations made by different users over different views of data (such as an EEG display with filters and transformations applied to the signal), to use provenance to reason about how the annotations relate to the original data, and to reason about the reliability and trustworthiness of each user's annotations over this data. To accomplish this, the research first defines data and provenance models that capture time- and space-varying data; novel reliability calculus algorithms for computing and dynamically updating the reliability and trustworthiness of individuals, based on their annotations and how these compare to annotations from recognized experts and the broader community; and a high-level language called PAL that enables the researchers to implement and compare multiple policies. The researchers will initially develop and validate the techniques on neuroscience and time series data, within a 900+ user public data sharing portal (with 1500+ EEG and other datasets for which annotations are required). The project team later expands the techniques to other data modalities, such as imaging and genomics