Continued Development and Maintenance of the MG-RAST Metagenomics Pipeline

 描述（由申请人提供）：宏基因组学是对直接从环境中采样的微生物种群的研究，为通过微生物分析发现新型酶提供了途径;使用微生物变化作为健康预测因子;或衡量人类活动的可持续性，如采矿。然而，宏基因组数据的量很大（例如，人类肠道微生物群的宏基因组的大小约为1个碱基对），并且需要进行的从大数据集中提取意义的处理是重要的，例如通过与不断更新的知识数据库进行比较来识别样本中的生物体基因组（分类注释）以及它们在做什么（功能注释）。这些数字只会随着实验者对宏基因组分析运行的需求越来越多而增长。出于这一需求，我们的MG-RAST（宏基因组学快速注释）门户网站，一个开源的，高通量的宏基因组学服务，自2008年以来一直是一个主要的社区资源，拥有超过16万个数据集和4万个用户。然而，自最初设计以来，MG-RAST见证了下一代测序技术的疯狂发展，极大地改变了计算环境（硬件和软件），改变了用户数量和数据集数量和多样性方面的需求，增加了管道组件的复杂性，以及对更高吞吐量的要求。为了适应这一点，MG-RAST一直在不断修改。修改包括升级管道组件，改进了几个算法;部署了一个定制的数据和工作流管理系统-SHOCK对象存储和AWE工作流管理器;并将MG-RAST移植到基于云的分布式架构。尽管我们不断地进行特别的系统改进，但我们的试点研究表明，需要对MG-RAST进行全面的重新设计，以跟上快速发展的宏基因组学领域的需求。我们提出的增强功能是基于表达的用户需求，新的使用模式，并灵活地将新的工具，特别是对查询序列的计算密集型相似性分析。通过这个项目，我们建议通过以下方式完成MG-RAST的转型：（i）提高其功能和数据再现性;（ii）通过自动监控和生成测试套件来提高其软件质量和性能;以及（iii）朝着宏基因组学数据的联合基础设施发展。总的来说，我们目标的成功实现将通过服务和数据的联合支持替代宏基因组学服务模型，并产生强大的最先进的宏基因组学资源。生物医学管道中的联合会通常是一个强大的方向，可以利用不同用户群的专业知识，并使其用户受益。因此，MG-RAST作为最先进的管道，将能够支持不断增长的用户群，处理更大和更多样化的数据集，并与新的基因组学技术协同发展。最终目标是加速最终用户应用的进步，例如，根据患者的微生物组量身定制的个性化药物。

 DESCRIPTION (provided by applicant): Metagenomics, the study of microbial populations sampled directly from the environment, affords avenues for discovering novel enzymes via microbial profiling; using microbial shifts as predictors for health; or gauging the sustainabilityof human operations like mineral mining. However, the volume of metagenomic data is large (e.g., the metagenome of a human's gut microbiota is about 1 Gigabasepairs in size) and the processing that needs to be done to extract meaning out of the large datasets is significant, such as to identify what organisms' genomes are in the sample (taxonomic annotation) and what are they doing (functional annotation) via comparisons with continually updated knowledge databases. These numbers are only growing as experimentalists demand more and more metagenomic analysis runs. Borne out of this need, our MG-RAST (Metagenomics-Rapid Annotation) portal, an open-source, high-throughput, metagenomics service, has been a major community resource since 2008, housing over 160K datasets and 40K users. However, since its original design, MG-RAST has witnessed the frenetic development of next-generation sequencing technologies, drastically altered computing landscape (both in hardware and software), changed requirements in terms of number of users and datasets' volumes and diversity, increasing complexity of pipeline components, and requirements for higher throughput. To adapt to this, MG-RAST has been continually modified. Modifications included upgrading the pipeline components with several algorithmic improvements; deploying a customized data and workflow management system - the SHOCK object store and AWE workflow manager; and porting MG-RAST to a cloud-based distributed architecture. Notwithstanding our continual, albeit ad-hoc system improvements, our pilot studies have indicated the need for a comprehensive redesign of MG-RAST to keep pace with the needs of the rapidly advancing field of metagenomics. Our proposed enhancements are based on expressed user requirements, new usage patterns, and flexibility to incorporate new tools, especially for the compute-intensive similarity analysis for queried sequences. Through this project, we propose to accomplish MG-RAST's transformation via (i) improving its functionality and data reproducibility; (ii) improving its software quality and performance through automated monitoring and generation of test suites; and (iii) moving toward a federated infrastructure for metagenomics data. Overall, the successful accomplishment of our aims will support alternate metagenomics service models through federation of services and data and result in a robust state-of-the-art metagenomics resource. Federation in biomedical pipelines is in general a powerful direction to leverage the expertise of diverse user-bases and, reciprocally, benefit its users. Thus, MG-RAST, as a state- of-the-art pipeline, will be capable of supporting an ever increasing user-base, handling larger and more varied datasets, and evolving in concert with new genomics technologies. This, with the ultimate goal, to accelerate advances in end-user applications, e.g., personalized medicine, tailored to the patient's microbiome.