CIF21 DIBBs: An Infrastructure Supporting Collaborative Data Analytics Workflow Design and Management

面对大数据的挑战，协作数据分析的需求显著增加。虽然工作流工具提供了一种定义、自动化和重复多步骤计算过程的正式方法，但设计复杂的数据处理工作流需要具有互补专业知识的多人协作。现有的工具不适合支持综合工作流的协同设计。为了应对这样的挑战，该项目旨在设计和开发一个能够支持面向协作数据的工作流组成和管理的软件基础设施，为现有的NSF网络基础设施增加一个关键组件，该网络基础设施将支持通过互联网的大数据协作。可重复性和可伸缩性是两个主要目标。该项目通过增加系统级设施来扩展现有的开源工作流工具VisTrails，以支持有效和高效的科学协作所必需的人类交互和协作。该项目将产生五个成果：1)配备图级种源查询形式主义的协作种源数据模型；2)解决格式转换的类型理论方法；3)基于超图理论的种源管理和挖掘算法；4)支持(A)同步协作科学工作流设计、合成、复制和可视化的软件工具；以及5)支持开发普遍适用的协作科学工作流合成工具的原则、方法、经验和教训。由此产生的工具将探索使用科学工作流加速需要在大数据分析方面进行协作的科学发现的潜力。这些工具的设计目标是土木工程学科中的用例，但有可能广泛影响科学和工程的其他领域。与VisTrails的合作伙伴关系使VisTrails最终用户社区能够使用和评估这些技术。

The need for collaborative data analysis increases significantly when confronted with the challenges of big data. Although workflow tools offer a formal way to define, automate, and repeat multi-step computational procedures, designing complex data process workflow requires collaboration from multiple people with complementary expertise. Existing tools are not suitable to support collaborative design of comprehensive workflows. To address such a challenge, this project aims to design and develop a software infrastructure with the capability of supporting collaborative data-oriented workflow composition and management, adding a key component to existing NSF cyberinfrastructure that will support big data collaboration through the Internet. Reproducibility and scalability are two major targets. The project extends an existing open-source workflow tool, VisTrails, by adding system-level facilities to support human interaction and cooperation that are essential for an effective and efficient scientific collaboration.This project will produce five outcomes:1) A collaborative provenance data model equipped with a graph-level provenance querying formalism;2) A type-theoretic approach for addressing format transformations;3) Hypergraph theory-based algorithms for provenance management and mining;4) A software tool supporting (a)synchronous collaborative scientific workflow design, composition, reproduction, and visualization; and 5) Principles, methodologies, experiences, and lessons that support the development of a generically applicable collaborative scientific workflow composition tool.The resulting tools will explore the potential for using scientific workflows to accelerate scientific discoveries that require a collaborative effort on big data analytics. The design of the tools is targeted toward use cases in the civil engineering discipline, but has the potential to broadly impact other areas of science and engineering. Partnership with VisTrails enables usage and evaluation of the techniques in the VisTrails end user community.