CC*DNI DIBBs: Data Analysis and Management Building Blocks for Multi-Campus Cyberinfrastructure through Cloud Federation

聚合、共享和分析重要大型数据集的能力，同时优化到科学的时间，对于支持多学科和多机构数据驱动的发现至关重要。该项目在纽约州和加利福尼亚州部署了一个联合云计算系统，该系统由数据基础设施构建块组成，旨在支持需要灵活工作流程和大规模数据集分析工具的科学家。来自七个不同领域的数据挑战——地球和大气科学、金融、化学、天文学、土木工程、基因组学和食品科学——正在使用一套丰富的开源软件、优化的框架和云使用模式来解决。联合云在康奈尔大学（项目负责人）以及布法罗大学和加州大学圣巴巴拉分校的合作站点运行。该项目团队正在支持拥有40多个全球合作者的多学科研究小组，并记录科学用例。该项目的更广泛目标是开发一个联合云模型，鼓励和奖励机构共享大规模数据分析资源，这些资源可以通过内部通用的增量构建块进行扩展，也可以通过与其他机构、公共云和NSF云资源的有意义的合作进行外部扩展。项目文档和网络研讨会介绍了最佳实践，包括如何创建虚拟机实例、在联合站点上运行、突发到Amazon Web Services，以及访问、移动和存储大规模数据。Open XDMoD （XD Net metrics on Demand）中内置了一个云度量的新工具，该工具以QBETS（时间序列队列边界估计）统计为特征，使用户能够在线预测未来的性能和分配级别可用性，并预测何时从联邦资源中爆发。新的分配和记帐模型允许机构管理员跟踪跨联邦站点的利用率，并将此数据用作交换机制。这些工具让我们更好地理解了跨机构边界共享数据基础设施构建块能力如何能够创造更广泛的科学和工程合作，并以可扩展和可持续的方式增加数据共享。

The ability to aggregate, share, and analyze important large data sets while optimizing time-to-science is essential to support multi-disciplinary and multi-institutional data-driven discovery. This project is deploying a federated cloud computing system in New York State and California comprised of data infrastructure building blocks designed to support scientists requiring flexible workflows and analysis tools for large-scale data sets. Data challenges from seven different communities-earth and atmospheric sciences, finance, chemistry, astronomy, civil engineering, genomics, and food science-are being addressed using a rich set of open source software, optimized frameworks, and cloud usage modalities. The federated cloud is operating at Cornell University (project lead) and at partner sites at the University at Buffalo and the University of California, Santa Barbara. The project team is supporting multi-disciplinary research groups with over forty global collaborators and documenting science use cases. The broader goal of this project is to develop a federated cloud model that encourages and rewards institutions for sharing large-scale data analysis resources that can be expanded internally with common, incremental building blocks and externally through meaningful collaborations with other institutions, public clouds, and NSF cloud resources.Project documentation and webinars feature best practices and include how to create Virtual Machine instances, run at federated sites, burst to Amazon Web Services, and access, move, and store large-scale data. A new tool for cloud metrics is being built into Open XDMoD (XD Net Metrics on Demand) that features QBETS (Queue Bounds Estimation from Time Series) statistics to enable users to make online forecasts of future performance and allocation level availability as well as to predict when to burst from federation resources. A new allocations and accounting model allows institutional administrators to track utilization across federated sites and use this data as an exchange mechanism. These tools provide a better understanding of how the sharing of data infrastructure building block capacity across institutional boundaries can create wider science and engineering collaborations and increase data sharing in a scalable and sustainable way.