CC*DNI DIBBs: Give Your Data the Edge: A Scalable Data Delivery Platform

点击翻译按钮获取中文摘要

Scientific collaboration is increasingly data driven. Large volumes of data are generated, aggregated, archived, and shared?often with collaborators (and their compute resources) spread across the globe - with subsequent analysis generating still more data to archive and distribute. Scientists either become experts at data transfer, storage, and management, or their ability to build on each other?s work suffers. This project addresses this challenge, with an emphasis on easing the burden on the user, building a solution that is sustainable over the long term, and delivering performance that scales in the number of collaborators. The project's technical approach is to deploy a general-purpose storage platform, called Syndicate, that harnesses a collection of available storage components to provide a global, scalable, and secure storage service. These include public and private cloud storage (for data durability), network caches and content distribution networks (for scalable read bandwidth), and local disks (for local reads/write performance). The project's goal is to make it possible for applications to access data independent of where it is stored, where Syndicate simultaneously: (1) minimizes the operational burden imposed on users, (2) maximizes the use of commodity infrastructure; and (3) maximizes aggregate I/O performance. At its core, Syndicate's value proposition is to fully decouple storage semantics from infrastructure. This lets users select infrastructure based on its cost/performance trade-off, while ensuring that their domain-specific storage requirements are met. And to demonstrate this value, the project is building a pilot deployment that spans edge resources on nine campuses, and operating the capability on behalf of a diverse set of application domains, ranging from biology to network analytics to personalized medicine to retail and consumer analysis.