2020BBSRC-NSF/BIO: REDEFINE - Development of efficient, large-scale metagenomics sequence comparison algorithms to facilitate novel genomic insights

微生物无处不在，在维持地球生命方面发挥着至关重要的作用，例如环境氧化、土壤养分循环以支持植物生长或促进动物消化。它们在植物和动物中引起许多疾病，并具有迅速进化以开发新的生态位和/或对抗抗菌剂的能力。宏基因组学是一个相对较新的领域，是一种独立于培养的方法，它应用复杂的DNA测序技术来分析来自任何环境的总微生物遗传物质。现在有可能重新组装数百万个短DNA序列，以产生样品中微生物基因组的表示，称为宏基因组组装基因组（MAGs），特别是对于细菌。虽然这种方法在计算上仍然很昂贵，但用于恢复这些基因组的计算机算法已经得到了实质性的改进，以提高mag的准确性。就在过去的五年里，许多大规模的研究，包括我们自己的研究，已经成功地应用了这些技术，累计产生了数百万个mag。这为科学家们提供了对99%尚未实验培养的生物体的新见解，并极大地扩展了生命之树。这些mag正在重塑我们对微生物群落结构和组成成员功能能力的理解。然而，MAG数量的爆炸式增长带来了新的挑战。这些大规模的分析可以产生与GenBank的大型基因组收集相匹配的基因组，这些基因组收集来自于对实验分离的微生物进行测序的传统技术。这样的基因组收集已经花了几十年的时间来建立，并由大型数据中心管理。然而，现在有必要在新的MAG集合和这样大的参考基因组集合之间进行常规比较。我们建议使用一种称为MinHash的特殊算法，它根据共享实体的数量快速估计两个集合之间的相似性，在我们的例子中是短序列。这种方法的大多数实现都集中在一个基因组与另一个基因组的快速比较。在本提案中，我们的目标是使用一系列计算技术来实现大型查询数据集与大型参考数据库的比较，其范围适用于微生物基因组，MAG集合和宏基因组序列。我们将开发并将此工具应用于一系列数据集，特别是那些位于MGnify（一个领先的宏基因组数据数据库）中的数据集。关键的应用是识别由计算方法引入的MAGs中的错误，通过识别数据集之间的重复MAGs来减少数据，将MAGs快速合并到特定环境中发现的基因组目录中，MAGs的分类分类（通过将相似距离转换为进化距离），以及分析元基因组数据集以确定可能发现哪些基因组。后一组配置文件还将能够描述MAG/基因组收集特征不佳的数据集，并优先考虑它们进行分析（即MAG生成）。这项建议的结果是多方面的。第一个是一套软件工具和相关的工作流，可以在计算机命令行上安装和运行。该工具的应用将导致多种新的数据输出（改进的MAGs，改进的目录和宏基因组剖面），这些数据将通过MGnify的网络界面提供。为了提供对这些MAG目录的快速访问，我们还将部署新的web界面（实现新的工具），允许用户将自己的MAG与已建立的收藏进行比较。这不仅将使科学研究民主化，而且还将减少对数据重复的需求。我们还将使用特定的用例来演示我们的工具的实用性，并为它们的使用提供培训和支持。

Microbes are ubiquitous and perform essential roles that help sustain life on earth, for e.g. environmental oxygenation, soil nutrient cycling to support plant growth or facilitating animal digestion. They cause many diseases in plants and animals and have the ability to rapidly evolve to exploit new niches and/or combat antimicrobials. A relatively new field, metagenomics is a culture independent method that applies sophisticated DNA sequencing technologies to analyse the total microbial genetic material from any environment. It is now possible to reassemble the millions of short DNA sequences to produce representations of the microbial genomes in a sample, termed metagenome assembled genomes (MAGs), especially for bacteria. While this approach remains computationally expensive, the computer algorithms used to recover these genomes have been substantially improved to increase accuracy of MAGs. Just in the past five years, many large-scale studies, including our own, have successfully applied these techniques to cumulatively generate millions of MAGs. This has provided scientists with novel insights into ~99% of organisms yet to be experimentally cultured and dramatically expanded the Tree of Life. These MAGs are reshaping our understanding of microbial community structure and the functional capacities of constituent members. This explosion in MAG numbers nevertheless presents new challenges. These large-scale analyses can generate genomes at magnitudes that match GenBank's large genome collection, which is derived from traditional techniques of sequencing experimentally isolated microbes. Such genome collections have taken decades to build and are managed by large data centres. Yet, there is now the need for groups to routinely perform comparisons between new MAG collections and such large reference genome collections. We propose to use a particular class of algorithm called MinHash, which rapidly estimates similarity between two sets based on the number of shared entities, in our case short sequences. Most implementations of this approach have focused on the rapid comparison of one genome to another. In this proposal, we aim to use a range of computational techniques to enable the comparison of a large query dataset to a large reference database, with the purview of being applied to microbial genomes, MAG collections and metagenomic sequences. We will develop and apply this tool to a range of datasets, particularly those housed in MGnify, a leading database of metagenomic data. The key applications are the identification of errors in MAGs which were introduced by the computational methods, data reduction by identifying duplicate MAGs between datasets, the rapid incorporation of MAGs into catalogues of genomes that have been found in a particular environment, taxonomic classification of MAGs (by converting similarity distances to evolutionary distances), and the profiling of metagenome datasets to determine which genomes are likely to be found. The latter set of profiles will also enable the delineation of datasets that are poorly characterised by MAG/genome collections and prioritise them for analysis (i.e. MAG generation). The outputs of this proposal are manifold. The first is a suite of software tools and associated workflows that can be installed and run on the computer command line. The application of the tool will lead to multiple new data outputs (refined MAGs, improved catalogues and metagenomic profiles) which will be made available via MGnify's web interfaces. To provide rapid access to these MAG catalogues, we will also deploy new web interfaces (implementing the new tools) that allow users to compare their own MAGs against established collections. This will not only democratise scientific research but also reduce the need for data duplication. We will also use specific use cases to demonstrate the utility of our tools and provide training and support for their use.