Developing novel deep-learning based methods for deciphering non-coding gene regulatory code

摘要 这个项目将贡献来自变形金刚的新的预先训练的DNA双向编码器表示， DNABERT，以及相关的深度学习工具，用于破译非编码DNA的语言并促进 从快速积累的序列数据中获得的基因调控信息与NLM的基因整合 数据库(例如，DBSNP、DBGaP和ClinVar)，通过以下方式为科学家和公共卫生提供服务 帮助识别疾病的遗传成分。而解释DNA如何翻译的遗传密码 转化为蛋白质是普遍存在的，决定基因表达时间和方式的调控密码各不相同 跨越不同的细胞类型和生物体。由于多义性的存在，非编码DNA高度复杂 和遥远的语义关系，从语言建模的角度来看。最近，深度学习方法已经 曾被用于解开基因调控代码，但未能在全球范围内有力地模拟这种语言 基因组中的特征，特别是在数据稀缺的情况下。为了应对这一挑战，我们提出了DNABERT 将DNA建模为一种语言，采用来自Transformers的双向编码器表示法的思想 (伯特)。根据最近在自然语言处理研究中的观察，我们假设预先训练的 基于变压器的神经网络模型提供了一种前景看好但尚未充分探索的深度学习方法 用于分析非编码DNA中的各种序列预测任务。我们的初步结果显示 人类基因组上的DNABERT在启动子和剪接位点上取得了最先进的表现 预测任务，在对小任务特定数据进行轻松微调后(Ji，Y.等人)。2020)。我们提议的目标是 研究是为各种序列预测任务开发DNABERT，并与现有状态进行基准测试- 最先进的基于深度学习的方法。具体目标是(1)通过适应发展新的深度学习方法 BERT；(2)将所提出的深度学习方法应用于特定的非编码DNA序列分析 和预测；以及(3)应用DNABERT预测和验证功能性非编码遗传变异 预测模型。拟议研究的一个主要贡献是开发了预先训练的DNABERT模型 以及预测算法，为DNA序列的分析和预测提供了新的强有力的方法。 由于DNABERT的预培训是资源密集型的，我们将提供源代码和预培训模型 为未来的学术研究做准备。我们还将开发一个集成的Web服务器，以(1)部署DNABERT 模型，(2)存储识别的序列特征和预测的数据库，以及(3)帮助用户 将DNABERT应用于他们特定的研究问题。我们期待DNABERT能够带来新的进步 并通过将先进的语言建模观点引入基因来为生物信息学社区提供见解 监管分析。

SUMMARY This project will contribute novel pre-trained DNA Bidirectional Encoder Representations from Transformers, called DNABERT, and associated deep-learning tools to decipher the language of non-coding DNA and facilitate integration of gene regulatory information from rapidly accumulating sequence data with NLM’s genetic databases (for example, dbSNP, dbGaP and ClinVar), which serve both scientists and the public health by helping identify the genetic components of disease. While the genetic code explaining how DNA is translated into proteins is universal, the regulatory code that determines when and how the genes are expressed varies across different cell-types and organisms. Non-coding DNA is highly complex due to the existence of polysemy and distant semantic relationship, from a language modeling perspective. Recently, deep learning methods have been used in unraveling the gene regulatory code, but failed to globally and robustly model such language features in the genome, especially in data-scarce scenarios. To address this challenge, we propose DNABERT to model DNA as a language, by adapting the idea of Bidirectional Encoder Representations from Transformers (BERT). Based on recent observations in natural language processing research, we hypothesize that pre-trained transformer-based neural network model offer a promising, and yet not fully explored, deep learning approach for a variety of sequence prediction tasks in the analysis of non-coding DNA. Our preliminary results showed that DNABERT on the human genome achieved state-of-the-art performance on promoter and splice-site prediction tasks, after easy fine-tuning on small task-specific data (Ji, Y. et al. 2020). The goal of our proposed research is to develop DNABERT for a variety of sequence prediction tasks, and benchmark with existing state- of-the-art deep-learning based methods. Specific aims are (1) develop novel deep-learning methods by adapting BERT; (2) apply the proposed deep-learning methods to specifically target non-coding DNA sequence analyses and predictions; and (3) predict and validate functional non-coding genetic variants by applying DNABERT prediction models. A major contribution of the proposed research is development of pre-trained DNABERT model and prediction algorithms, which present new powerful methods for analyses and predictions of DNA sequences. Since the pre-training of DNABERT is resource-intensive, we will provide the source code and pre-trained model at Github for future academic research. We will also develop an integrated web server to (1) deploy DNABERT model, (2) database to store the identified sequence features and predictions, and (3) tutorials to help users to apply DNABERT to their specific research problems. We anticipate that DNABERT can bring new advancements and insights to the bioinformatics community by bringing advanced language modeling perspective to gene regulation analyses.