CIF21 DIBBs: An Infrastructure Supporting Collaborative Data Analytics Workflow Design and Management

面对大数据的挑战，协作数据分析的需求显著增加。尽管工作流工具提供了一种正式的方式来定义、自动化和重复多步计算过程，但设计复杂的数据处理工作流需要来自具有互补专业知识的多个人员的协作。现有的工具不适合支持全面工作流的协作设计。为了应对这一挑战，该项目旨在设计和开发一个软件基础设施，该基础设施具有支持协作数据导向工作流组合和管理的能力，为现有的NSF网络基础设施添加一个关键组件，以支持通过互联网进行大数据协作。再现性和可伸缩性是两个主要目标。该项目扩展了现有的开源工作流工具VisTrails，通过增加系统级的设施来支持人类交互和合作，这对于有效和高效的科学协作是必不可少的。该项目将产生五个成果：1)一个具有图级来源查询形式的协同来源数据模型；2)处理格式转换的类型理论方法；3)基于超图理论的溯源管理与挖掘算法；4)支持(A)同步协同科学工作流设计、组成、复制和可视化的软件工具；5)支持开发通用的协作科学工作流组合工具的原则、方法、经验和教训。由此产生的工具将探索利用科学工作流程加速需要大数据分析协作的科学发现的潜力。这些工具的设计是针对土木工程学科中的用例的，但也有可能广泛地影响科学和工程的其他领域。与VisTrails的合作伙伴关系支持在VisTrails最终用户社区中使用和评估这些技术。

The need for collaborative data analysis increases significantly when confronted with the challenges of big data. Although workflow tools offer a formal way to define, automate, and repeat multi-step computational procedures, designing complex data process workflow requires collaboration from multiple people with complementary expertise. Existing tools are not suitable to support collaborative design of comprehensive workflows. To address such a challenge, this project aims to design and develop a software infrastructure with the capability of supporting collaborative data-oriented workflow composition and management, adding a key component to existing NSF cyberinfrastructure that will support big data collaboration through the Internet. Reproducibility and scalability are two major targets. The project extends an existing open-source workflow tool, VisTrails, by adding system-level facilities to support human interaction and cooperation that are essential for an effective and efficient scientific collaboration.This project will produce five outcomes:1) A collaborative provenance data model equipped with a graph-level provenance querying formalism;2) A type-theoretic approach for addressing format transformations;3) Hypergraph theory-based algorithms for provenance management and mining;4) A software tool supporting (a)synchronous collaborative scientific workflow design, composition, reproduction, and visualization; and 5) Principles, methodologies, experiences, and lessons that support the development of a generically applicable collaborative scientific workflow composition tool.The resulting tools will explore the potential for using scientific workflows to accelerate scientific discoveries that require a collaborative effort on big data analytics. The design of the tools is targeted toward use cases in the civil engineering discipline, but has the potential to broadly impact other areas of science and engineering. Partnership with VisTrails enables usage and evaluation of the techniques in the VisTrails end user community.