Collaborative Research: The CrystalGrid Framework

CrystalGrid框架（CGF）项目将研究X射线晶体学领域整个数据空间的数据采集、传输和管理，解决管理数据表示、格式、数据容器、管理域和各种仪器和设备的广泛异质性的方法。直到最近，个别实验室只是强加了格式和程序的本地同质性，而不是存储依赖于实验室的元数据。然而，由于晶体学家开始在实验室之间交叉以实现其研究目标，并且由于输出数据流的数量和大小的增加使得每次研究的时间减少，因此这种特设系统是有限的。本地工作流程必须明确，过程必须正式描述，历史和数据组合必须以开放、可共享的方式表达。为每个实验创建和管理完整的、可访问的记录至关重要，在整个现场的数据采集和管理中也是如此。为了满足这一需求，该项目将开发一个网络服务接口和数据和元数据系统的框架，解决整个晶体学的频谱。项目参与者和合作者将利用现有的项目，如互惠网和通用仪器中间件架构，解决问题领域中的较窄问题。CGF还将利用具有重叠感兴趣领域的合作项目，例如英国的Comb-e-Chem项目。CGF项目涉及处理晶体学应用空间中数据、程序和仪器的异质性的经典问题，以及整合整个数据收集、传输、存储和存储的另一个经典问题。和管理需求的领域到一个无缝的开始到结束的系统。挑战在于创建一个虚拟化系统，该系统在多个方面管理异构性，并仅使用开放的、可扩展的和可互操作的标准和方法来提供垂直集成。执行研究的计划以生产产品为中心（CGF）这将立即有助于解决X射线晶体学领域中出现的技术问题。在晶体学中，一个具体的目标是使结构结果可以获得，否则可能永远不会看到，因此CGF将有助于增加科学知识的主体，并提高联邦投资的回报率在全国范围内的大量X射线衍射仪和相关仪器。尽管该项目专门针对全球数百个晶体学实验室，但其中创建的软件和方法旨在可重复使用于从单个实验室实践到共享的全球合作实验室系统的任何科学。在高能物理和天文学等科学领域，科学家们长期以来一直共享单一、独特的大型仪器，并不得不创建共享的数据管理和仪器元数据。CGF可能在其他科学学科中有用，这些学科仍然使用广泛分布的实验室仪器，现在需要在数据网格中连接。

The CrystalGrid Framework (CGF) project will research the acquisition, transport, and curation of data over the entire data space of the field of X-ray crystallography, addressing methods for managing wide heterogeneity in data representations, formats, data containers, administrative domains and diverse instruments and equipment. Until recently, individual labs have simply imposed local homogeneity of format and procedure, and not stored lab-dependent metadata. This ad hoc system is limited, however, as crystallographers begin to cross between labs to accomplish their research objectives, and as increasing numbers and sizes of output data streams leave less time for each investigation. Local workflow must be made explicit, procedures must be formally described, and the history and assemblages of data expressed in an open, shareable way. Creation and management of complete, accessible records for each experiment is critical, as well as heterogeneity in data acquisition and management across the field. To meet that need, this project will develop a framework of web service interfaces and data and metadata systems addressing the whole spectrum of crystallography. Project participants and collaborators will leverage existing projects, such as Reciprocal Net and Common Instrument Middleware Architecture, that address narrower issues in the problem domain. The CGF will also draw on collaborating projects with overlapping areas of interest, such as the UK-based Comb-e-Chem project. The resulting framework will be a useful environment for crystallographic investigations and an extensible platform on which new web-based applications can be built.The CGF project involves the classic problem of dealing with heterogeneity in data, procedures, and instruments in the crystallography application space, and another classic problem in integrating the entire data collection, transport, and curation requirements of the domain into a seamless beginning to end system. The challenge is to create a virtualization system that manages heterogeneity in more than a single aspect and to provide vertical integration using only open, extensible, and interoperable standards and methodologies.While the project constitutes research into pertinent computer science problems, the plan for performing the research is centered on producing a product (the CGF) that will immediately be useful in addressing emerging technical problems in the field of X-ray crystallography. Within crystallography, one of the specific goals is to make structural results accessible that might otherwise never be seen, and so the CGF will help increase the body of scientific knowledge and improve the return on federal investment in the large numbers of x-ray diffractometers and associated instruments nationwide. Although the project targets specifically a few hundreds of crystallography labs worldwide, the software and methods created in it are intended to be reusable for any science moving from individual lab practices to a shared, global collaboratory system. In sciences such as high-energy physics and astronomy, the scientists have long shared single, unique, large instruments and had to create shared data management and instrument metadata. CGF is likely to be useful in other scientific disciplines which still use widely-distributed lab-based instruments that now need to be linked in data grids.