Data CI Pilot: VariMat Streaming Polystore Integration of Varied Experimental Materials Data

VariMat是一个试点项目，旨在整合实验数据源与网络基础设施组件，弥合实验材料研究数据的数据多样性差距。材料科学与工程的中心目标是发现和利用创新材料为社会服务，但这个过程是复杂的，往往是缓慢的。“材料基因组计划”（materials Genome Initiative， MGI）最近的工作重点是加速新材料的发现，该计划认识到能源、交通和国家安全等领域对新材料的迫切需求。MGI的核心是利用数据革命，推动包括人工智能和机器学习在内的数据密集型方法。然而，为了发挥MGI的潜力，迫切需要强大的、高性能的数据网络基础设施（CI），以促进机器可操作的数据和更好地实施适合材料领域的FAIR数据原则。在剩余的CI缺口中，最重要的是需要整合实验数据，使其更具可操作性并符合研究界的需求。这种复杂的差距由于材料科学和工程的跨学科性质而变得更加复杂，其中大多数项目依赖于由高度多样化的技术、分布的实验室和多个研究人员收集的实验数据。这些数据层的多样性和数量与材料研究的分散性相结合，造成了数据多样性的差距，阻碍了许多数据饥渴型机器学习方法所需的数据的快速使用和有价值的重用。VariMat Data CI旨在通过在量子材料子领域的试点实例来打破这些障碍，并在其整个生命周期中最大化实验数据价值。该项目将UCSB量子铸造厂和paradigm（材料创新平台）的团队联系在一起，这是NSF在量子飞跃方面的两项主要投资。这种联系将强大的科学驱动因素与基础设施发展结合起来，同时采取战略重点，关注有影响力的高质量、大批量数据生产中心，这些中心是用户培训和工作流程的渠道。VariMat将为研究人员提供综合、及时的实验数据，以实现新的发现途径，并推动依赖于可控、可复制合成的新材料的开发；结构和成分表征；财产的决心;以及与理论和建模研究的关联性。提出的研究将为集成数据基础设施建立一个新的范例，利用流层实时摄取到多存储。VariMat使用自动化流层将仪器数据链接到异构数据管理系统的多存储库，该系统可优化不同数据类型的存储、查询和访问。polystore包含多个数据模型，同时为用户统一查询过程。VariMat polystore为统一管理和查询跨分布式设施创建的不同实验大数据提供了新的选择，并扩展了材料领域的FAIR数据合规性。VariMat实现了一个创新的流处理数据输入模块，用于分析驱动的实验数据摄取。流层和多存储层共同服务于一个面向用户的门户网站，它将高级搜索与数据分析、可视化和计算资源结合在一起。这种集成将通过特定于实例化且跨越整个项目的统一语义标准来促进。VariMat将利用PARADIM数据模型，用有向无环图（DAG）描述合成和表征，允许遍历材料的整个历史。VariMat的“松散耦合”体系结构提供了子系统的操作和管理独立性，非常适合地理上分布的系统，这些系统在组件中不断发展，这在中等规模或较大的材料科学研究中是典型的。虽然基础设施填补了一个关键的、社区确定的空白，但VariMat组件将很容易部署，并在依赖于分布式、操作独立的仪器实验室的其他科学领域具有广泛的适用性。自动化部署和开源组件将促进新领域中的实例化。为了最大限度地发挥影响，开发的概念和工具将通过免费提供、开源代码、在线教程和数据集以及利用量子铸造厂和paradigm现有学校和讲习班的培训来传播。该奖项由先进网络基础设施办公室颁发，由美国国家科学基金会数学和物理科学理事会材料研究部联合支持。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

VariMat is a pilot project to integrate experimental data sources with cyberinfrastructure components and bridge the data variety gap in experimental materials research data. The central goal of Materials Science and Engineering is to discover and deploy innovative materials to serve society, but the process is complex and frequently slow. Recent work to accelerate such materials discovery is focused by the Materials Genome Initiative (MGI), which recognizes the critical need for new materials in fields as diverse as energy, transportation, and national security. The MGI centers on harnessing the data revolution and fueling data-intensive methods including artificial intelligence and machine learning. To reach the potential of the MGI, however, there is pressing need for robust, high-performance data cyberinfrastructure (CI) that facilitates machine-actionable data and better implementation of FAIR data principles suited to the materials domain. Among remaining CI gaps, none is more important than the need to integrate experimental data in ways that make it more operable and consonant to the research community needs. This complex gap is compounded by the transdisciplinary nature of materials science and engineering where most projects depend on experimental data collected by highly varied techniques, in distributed labs, and by multiple investigators. The variety and volume of this data layers onto the dispersed nature of materials research to create a data variety gap that impedes both rapid use and valuable reuse of data as required by many data-hungry machine learning methods. The VariMat Data CI is designed to break these barriers with a pilot instantiation in the subdomain of quantum materials and maximize experimental data value across its whole lifecycle. The project links teams from the UCSB Quantum Foundry and PARADIM, a Materials Innovation Platform (MIP) – two of the NSF's premier investments in the Quantum Leap. This linkage integrates strong science drivers with infrastructure development while adopting a strategic focus on influential centers of high-quality, high-volume data production that are conduits to user training and workflows. VariMat will provide investigators with integrated and timely access to the breadth of experimental data needed to enable new discovery pathways and drive novel-materials development that relies on controlled, replicable synthesis; structural and compositional characterization; property determination; and connectability to theory and modeling studies.The proposed research will establish a new paradigm for an integrated data infrastructure leveraging a streaming layer for real-time ingest to a polystore. VariMat uses an automated streaming layer to link instrumental data to a polystore of heterogeneous data management systems that optimize storage, query, and access for disparate data types. The polystore encompasses multiple data models while unifying the query process for users. The VariMat polystore creates a new option for unified management and query of disparate experimental Big Data created across distributed facilities and to expand FAIR data compliance in the materials domain. VariMat implements an innovative stream processing Data Ingress Module for analytics driven ingest of experimental data. Together, the streaming and polystore layers serve a user-oriented web portal that combines advanced search with data analysis, visualization, and compute resources. Such integration will be facilitated by a unified semantic standard specific to the instantiation and that spans the project. VariMat will leverage the PARADIM data model describing synthesis and characterization with a directed acyclic graph (DAG) allowing traversal of the materials entire history. VariMat's "loosely coupled" architecture provides operational and managerial independence of subsystems well suited for geographically distributed systems with on-going evolution in components as is typical in mid-scale or larger materials science research. While the infrastructure fills a critical, community identified gap, VariMat components will be readily deployable and have broad applicability in other scientific fields dependent on distributed, operationally independent instrumental laboratories. Automated deployment and open source components will facilitate ready instantiation in new domains. To maximize impact, concepts and tools developed will be disseminated through freely available, open source codes, online tutorials and data sets, and trainings that leverage existing schools and workshops at the Quantum Foundry and PARADIM.This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Division of Materials Research within the NSF Directorate for Mathematical and Physical Sciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.