2020BBSRC-NSF/BIO: REDEFINE - Development of efficient, large-scale metagenomics sequence comparison algorithms to facilitate novel genomic insights

微生物无处不在，发挥着帮助维持地球上的生命的基本作用，例如环境氧化、土壤养分循环以支持植物生长或促进动物消化。它们会导致动植物中的许多疾病，并有能力迅速进化，以开发新的利基和/或对抗抗菌剂。元基因组学是一个相对较新的领域，是一种独立于文化的方法，它应用复杂的DNA测序技术来分析任何环境中的总微生物遗传物质。现在有可能重组数百万个短DNA序列，在一个样本中产生微生物基因组的代表，称为元基因组组装基因组(MAG)，特别是对细菌来说。虽然这种方法在计算上仍然很昂贵，但用于恢复这些基因组的计算机算法已经得到了实质性的改进，以提高MAG的准确性。就在过去的五年里，包括我们自己在内的许多大规模研究已经成功地应用这些技术累积产生了数百万个磁珠。这为科学家们提供了对~99%尚未进行实验培养的生物体的新见解，并极大地扩大了生命树。这些MAG正在重塑我们对微生物群落结构和组成成员功能能力的理解。然而，MAG数量的爆炸式增长带来了新的挑战。这些大规模的分析可以产生与GenBank的大型基因组收集相匹配的基因组，这些基因组收集来自传统的对实验分离的微生物进行测序的技术。这样的基因组收集花了几十年的时间建立起来，并由大型数据中心管理。然而，现在有必要让研究小组定期对新的MAG样本和如此庞大的参考基因组样本进行比较。我们建议使用一种特殊的称为MinHash的算法，它基于共享实体的数量快速估计两个集合之间的相似性，在我们的例子中是短序列。这种方法的大多数实施都集中在快速比较一个基因组与另一个基因组。在这个建议中，我们的目标是使用一系列计算技术来实现大型查询数据集与大型参考数据库的比较，目的是将其应用于微生物基因组、MAG集合和元基因组序列。我们将开发这一工具并将其应用于一系列数据集，特别是那些保存在MGnify中的数据集，MGnify是一个领先的元基因组数据数据库。关键的应用是查明由计算方法引入的MAG中的错误，通过识别数据集之间的重复MAG来减少数据，将MAG快速并入已在特定环境中发现的基因组目录，MAG的分类分类(通过将相似距离转换为进化距离)，以及元基因组数据集的概况以确定可能发现哪些基因组。后一组配置文件还将能够描述MAG/基因组集合特征不佳的数据集，并将其优先用于分析(即MAG生成)。这项建议的成果是多方面的。第一个是可以在计算机命令行上安装和运行的一套软件工具和相关的工作流程。该工具的应用将产生多个新的数据输出(改进的MAG、改进的目录和元基因组概况)，这些数据将通过MGnify的网络界面提供。为了提供对这些MAG目录的快速访问，我们还将部署新的网络界面(实施新工具)，允许用户将他们自己的MAG与已建立的收藏进行比较。这不仅将使科学研究民主化，还将减少数据重复的需要。我们还将使用特定的用例来演示我们的工具的实用性，并为它们的使用提供培训和支持。

Microbes are ubiquitous and perform essential roles that help sustain life on earth, for e.g. environmental oxygenation, soil nutrient cycling to support plant growth or facilitating animal digestion. They cause many diseases in plants and animals and have the ability to rapidly evolve to exploit new niches and/or combat antimicrobials. A relatively new field, metagenomics is a culture independent method that applies sophisticated DNA sequencing technologies to analyse the total microbial genetic material from any environment. It is now possible to reassemble the millions of short DNA sequences to produce representations of the microbial genomes in a sample, termed metagenome assembled genomes (MAGs), especially for bacteria. While this approach remains computationally expensive, the computer algorithms used to recover these genomes have been substantially improved to increase accuracy of MAGs. Just in the past five years, many large-scale studies, including our own, have successfully applied these techniques to cumulatively generate millions of MAGs. This has provided scientists with novel insights into ~99% of organisms yet to be experimentally cultured and dramatically expanded the Tree of Life. These MAGs are reshaping our understanding of microbial community structure and the functional capacities of constituent members. This explosion in MAG numbers nevertheless presents new challenges. These large-scale analyses can generate genomes at magnitudes that match GenBank's large genome collection, which is derived from traditional techniques of sequencing experimentally isolated microbes. Such genome collections have taken decades to build and are managed by large data centres. Yet, there is now the need for groups to routinely perform comparisons between new MAG collections and such large reference genome collections. We propose to use a particular class of algorithm called MinHash, which rapidly estimates similarity between two sets based on the number of shared entities, in our case short sequences. Most implementations of this approach have focused on the rapid comparison of one genome to another. In this proposal, we aim to use a range of computational techniques to enable the comparison of a large query dataset to a large reference database, with the purview of being applied to microbial genomes, MAG collections and metagenomic sequences. We will develop and apply this tool to a range of datasets, particularly those housed in MGnify, a leading database of metagenomic data. The key applications are the identification of errors in MAGs which were introduced by the computational methods, data reduction by identifying duplicate MAGs between datasets, the rapid incorporation of MAGs into catalogues of genomes that have been found in a particular environment, taxonomic classification of MAGs (by converting similarity distances to evolutionary distances), and the profiling of metagenome datasets to determine which genomes are likely to be found. The latter set of profiles will also enable the delineation of datasets that are poorly characterised by MAG/genome collections and prioritise them for analysis (i.e. MAG generation). The outputs of this proposal are manifold. The first is a suite of software tools and associated workflows that can be installed and run on the computer command line. The application of the tool will lead to multiple new data outputs (refined MAGs, improved catalogues and metagenomic profiles) which will be made available via MGnify's web interfaces. To provide rapid access to these MAG catalogues, we will also deploy new web interfaces (implementing the new tools) that allow users to compare their own MAGs against established collections. This will not only democratise scientific research but also reduce the need for data duplication. We will also use specific use cases to demonstrate the utility of our tools and provide training and support for their use.