Data CI Pilot: VariMat Streaming Polystore Integration of Varied Experimental Materials Data

VariMat是一个试点项目，旨在将实验数据源与网络基础设施组件相集成，并弥合实验材料研究数据中的数据多样性差距。材料科学与工程的中心目标是发现和部署创新材料，以服务社会，但这一过程是复杂的，往往缓慢。材料基因组计划（MGI）最近致力于加速此类材料的发现，该计划认识到能源、交通和国家安全等领域对新材料的迫切需求。 MGI专注于利用数据革命，并推动数据密集型方法，包括人工智能和机器学习。 然而，为了发挥MGI的潜力，迫切需要强大的高性能数据网络基础设施（CI），以促进机器可操作的数据和更好地实施适合材料领域的FAIR数据原则。 在剩下的CI差距，没有比需要整合实验数据的方式，使其更具可操作性和和谐的研究社区的需求更重要。 材料科学和工程的跨学科性质加剧了这一复杂的差距，其中大多数项目依赖于通过高度多样化的技术、分布式实验室和多个研究人员收集的实验数据。这些数据的多样性和数量叠加在材料研究的分散性上，造成了数据多样性的差距，阻碍了许多数据饥渴的机器学习方法所需的数据的快速使用和有价值的重复使用。VariMat Data CI旨在通过量子材料子域中的试点实例化来打破这些障碍，并在其整个生命周期中最大限度地提高实验数据价值。 该项目将UCSB Quantum Foundry和材料创新平台（MIP）的PARADIM团队联系起来，这是NSF在Quantum Leap中的两项主要投资。这种联系将强大的科学驱动力与基础设施发展相结合，同时采取战略重点，关注高质量，大批量数据生产的有影响力的中心，这些中心是用户培训和工作流程的管道。 VariMat将为研究人员提供集成和及时访问所需的实验数据的广度，以实现新的发现途径，并推动依赖于受控，可复制合成的新型材料开发;结构和成分表征;属性确定;以及与理论和建模研究的可连接性。拟议的研究将为集成数据基础设施建立一个新的范例，该基础设施利用流层进行真实的时间摄入到polystore。 VariMat使用自动流层将仪器数据链接到异构数据管理系统的聚合存储，优化不同数据类型的存储，查询和访问。 polystore包含多个数据模型，同时统一了用户的查询过程。VariMat polystore为统一管理和查询跨分布式设施创建的不同实验大数据创建了一个新选项，并扩展了材料领域的FAIR数据合规性。VariMat实现了一个创新的流处理数据入口模块，用于分析驱动的实验数据摄取。流层和polystore层一起提供面向用户的Web门户，该门户将高级搜索与数据分析、可视化和计算资源相结合。这样的集成将通过一个统一的语义标准来促进，该标准特定于实例化并跨越项目。VariMat将利用PARADIM数据模型，通过有向无环图（DAG）描述合成和表征，允许遍历材料的整个历史。 VariMat的“松散耦合”架构提供了子系统的操作和管理独立性，非常适合地理上分布式的系统，组件正在进行演变，这在中等规模或更大的材料科学研究中是典型的。 虽然基础设施填补了一个关键的，社区确定的差距，VariMat组件将随时部署，并在其他科学领域依赖于分布式，操作独立的仪器实验室具有广泛的适用性。 自动化部署和开源组件将促进新领域中的现成实例化。为了最大限度地扩大影响，将通过免费提供的开放源码、在线教程和数据集传播所开发的概念和工具，该奖项由高级网络基础设施办公室颁发，并得到NSF数学和物理科学理事会材料研究部的共同支持。该奖项反映了NSF的法定使命并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

VariMat is a pilot project to integrate experimental data sources with cyberinfrastructure components and bridge the data variety gap in experimental materials research data. The central goal of Materials Science and Engineering is to discover and deploy innovative materials to serve society, but the process is complex and frequently slow. Recent work to accelerate such materials discovery is focused by the Materials Genome Initiative (MGI), which recognizes the critical need for new materials in fields as diverse as energy, transportation, and national security. The MGI centers on harnessing the data revolution and fueling data-intensive methods including artificial intelligence and machine learning. To reach the potential of the MGI, however, there is pressing need for robust, high-performance data cyberinfrastructure (CI) that facilitates machine-actionable data and better implementation of FAIR data principles suited to the materials domain. Among remaining CI gaps, none is more important than the need to integrate experimental data in ways that make it more operable and consonant to the research community needs. This complex gap is compounded by the transdisciplinary nature of materials science and engineering where most projects depend on experimental data collected by highly varied techniques, in distributed labs, and by multiple investigators. The variety and volume of this data layers onto the dispersed nature of materials research to create a data variety gap that impedes both rapid use and valuable reuse of data as required by many data-hungry machine learning methods. The VariMat Data CI is designed to break these barriers with a pilot instantiation in the subdomain of quantum materials and maximize experimental data value across its whole lifecycle. The project links teams from the UCSB Quantum Foundry and PARADIM, a Materials Innovation Platform (MIP) – two of the NSF's premier investments in the Quantum Leap. This linkage integrates strong science drivers with infrastructure development while adopting a strategic focus on influential centers of high-quality, high-volume data production that are conduits to user training and workflows. VariMat will provide investigators with integrated and timely access to the breadth of experimental data needed to enable new discovery pathways and drive novel-materials development that relies on controlled, replicable synthesis; structural and compositional characterization; property determination; and connectability to theory and modeling studies.The proposed research will establish a new paradigm for an integrated data infrastructure leveraging a streaming layer for real-time ingest to a polystore. VariMat uses an automated streaming layer to link instrumental data to a polystore of heterogeneous data management systems that optimize storage, query, and access for disparate data types. The polystore encompasses multiple data models while unifying the query process for users. The VariMat polystore creates a new option for unified management and query of disparate experimental Big Data created across distributed facilities and to expand FAIR data compliance in the materials domain. VariMat implements an innovative stream processing Data Ingress Module for analytics driven ingest of experimental data. Together, the streaming and polystore layers serve a user-oriented web portal that combines advanced search with data analysis, visualization, and compute resources. Such integration will be facilitated by a unified semantic standard specific to the instantiation and that spans the project. VariMat will leverage the PARADIM data model describing synthesis and characterization with a directed acyclic graph (DAG) allowing traversal of the materials entire history. VariMat's "loosely coupled" architecture provides operational and managerial independence of subsystems well suited for geographically distributed systems with on-going evolution in components as is typical in mid-scale or larger materials science research. While the infrastructure fills a critical, community identified gap, VariMat components will be readily deployable and have broad applicability in other scientific fields dependent on distributed, operationally independent instrumental laboratories. Automated deployment and open source components will facilitate ready instantiation in new domains. To maximize impact, concepts and tools developed will be disseminated through freely available, open source codes, online tutorials and data sets, and trainings that leverage existing schools and workshops at the Quantum Foundry and PARADIM.This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Division of Materials Research within the NSF Directorate for Mathematical and Physical Sciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.