Enhancing gnomAD Sustainability: Implementing Site Reliability Engineering Principles for Genomic Data Infrastructure

点击翻译按钮获取中文摘要

Project Summary The gnomAD (Genome Aggregation Database) project is a large-scale effort to aggregate and harmonize exome and genome sequencing data from diverse human populations, aiming to provide a comprehensive resource to identify and interpret genetic variation in the human genome. gnomAD has become a critical resource for scientists worldwide, enabling them to study the impact of genetic variation on human health and disease. The parent U24 award supports the gnomAD project by providing funding to (1) aggregate and produce increasingly large datasets, (2) integrate gnomAD with other datasets, resources, and statistical tools, and (3) disseminate the results to the wider research community. The parent award ensures the continued development of a high-quality, variant database for researchers, empowering them to advance our understanding of human genetics and contribute to the development of novel diagnostics and therapeutics. This administrative supplement proposes to strengthen the gnomAD project's longevity by incorporating Site Reliability Engineering (SRE) principles and practices. SRE is a new engineering discipline that was invented at large technology companies to streamline management of computational infrastructure, but has not yet seen widespread adoption in academia. The proposal's specific aims are twofold: the first aim involves developing SRE practices within the gnomAD project by refactoring the existing codebase. This will improve the project’s ability to meet unique challenges related to increasing data scale, manage computational environments for the production/research teams, and meet users' expectations for a highly performant browser resource. The second aim relates to creating high quality public documentation for our solutions and publishing these in tool registries for other genomics projects to use, copy, and repurpose.