SDCI Data Improvement: Data Grids for Community Driven Applications

OCI 0721400Reagan Moore加州大学圣迭戈分校SDCI数据改进：社区驱动应用程序的数据网格通过创建共享数据集合促进学术机构之间的协作研究。分布式数据管理的事实标准是由圣地亚哥超级计算机中心(SDSC)的数据密集型计算环境小组开发的存储资源代理数据网格(SRB)。SRB被广泛用于国家和国际项目，以支持用于共享数据的数据网格、用于发布和发现数据的数字图书馆、用于保存数据的永久档案以及用于联合传感器数据的实时系统。SRB是通用的网络基础设施(软件中间件)，用于生产管理天文学、生物学、生态学、高能物理、医学、海洋学、地震学和人文科学等科学领域超过2 PB的数据。智力优势SRB通过抽象数据共享、发现和访问所需的主要对象(文件、流、元数据、资源、用户)来提供关键的网络基础设施。根据开发SRB的经验和来自强大用户社区的反馈，我们发现还需要对数据管理策略进行抽象。为了自动化管理策略的执行，我们正在开发下一代分布式数据管理技术，称为集成的面向规则的数据系统(IRODS)。该系统将基于规则的数据管理与服务器端工作流技术相结合，以执行微服务(执行明确定义的任务的软件模块)。将管理策略描述为控制微服务执行的规则的能力使得首次能够定义特定于用户和社区范围的策略，自动化管理策略的执行，并验证关于存储库管理的断言。IRODS背后的概念为开发可定制和可验证的中间件系统提供了一个平台。基于规则的系统结合了来自受保护的逻辑程序(正向链接和回溯)、数据库(触发器和ECA规则)和抽象工作流系统的思想和技术。过程性和声明性语义控制模块化软件，不仅用于定制，还用于程序验证。微服务的声明性链接提供的模块化支持远程操作执行的基于规则的验证。管理虚拟化的创新应用支持数据网格、数字图书馆、永久存储库和档案以及教育环境的政策执行自动化，预计这些环境将为21世纪的许多科学突破做出贡献。新方法使支持分布式数据管理的网络基础设施的所有组件能够有计划地发展。这项提议有三个目标：维持对SRB数据管理软件的支持；开发和支持下一代基于规则的数据网络基础设施；制定过渡框架，以便迁移到新的数据管理基础设施。其目的是开发可演变的解决方案，将国家网络基础设施的全部力量带给科学家和工程师社区。更广泛的影响SRB软件支持跨学科数据共享以及特定学科协作。IRODS软件中数据管理策略的定制和抽象功能将帮助小型和大型社区使用统一的网络基础设施共享数据。

OCI 0721400Reagan MooreUniversity of California at San DiegoSDCI Data Improvement: Data Grids for Community Driven Applications Collaborative research between academic institutions is facilitated through the creation of shared data collections. The de facto standard for distributed data management is the Storage Resource Broker data grid (SRB) developed by the Data Intensive Computing Environment group at the San Diego Supercomputer Center (SDSC). The SRB is used extensively in national and international projects in support of data grids for sharing data, digital libraries for publishing and discovering data, persistent archives for preserving data, and real-time systems for federating sensor data. The SRB is generic cyberinfrastructure (software middleware) that is used in production managing over 2 PBs of data across scientific domains such as astronomy, biology, ecology, high energy physics, medicine, oceanography, seismology, and the humanities. Intellectual merit The SRB provides critical cyberinfrastructure by abstracting the main objects (files, streams, metadata, resources, users) needed for data sharing, discovery and access. Based on the experience in developing the SRB and feedback from a robust user community, we have found that abstraction of data management policies is also needed. To automate execution of management policies, we are developing the next generation of distributed data management technology called the integrated Rule Oriented Data System (iRODS). The system integrates rule-based data management with server-side workflow technology for executing micro-services (software modules that perform well-defined tasks). The ability to characterize management policies as rules controlling the execution of micro-services makes it possible for the first time to define both user-specific and community-wide policies, automate the execution of the management policies, and validate assertions about repository management. The concepts behind iRODS provide a platform for developing customizable and verifiable middleware systems. The rule based system combines ideas and techniques from guarded logic programs (forward-chaining and backtracking), databases (triggers and ECA rules), and abstract workflow systems. Procedural and declarative semantics control modular software for not only customization but also program verification. The modularity provided by declarative chaining of micro-services enables rule-based validation of remote operation execution. The innovative application of management virtualization supports automation of policy execution for data grids, digital libraries, persistent repositories and archives, as well as educational environments that are expected to contribute to many of the scientific breakthroughs of the 21st century. The new approach enables planned evolution of all components of cyberinfrastructure that support distributed data management. This proposal has three aims: to maintain support for the SRB data management software; to develop and support the next generation of rule-based data cyber-infrastructure and to develop a transition framework to migrate to the new data management infrastructure. The aim is to develop evolvable solutions to bring the full power of a national cyberinfrastructure to communities of scientists and engineers. Broader Impact The SRB software enables inter-disciplinary data sharing as well as discipline specific collaborations. The customization and abstraction features of data management policies in the iRODS software will help both small and large communities share data using uniform cyberinfrastucture.