Deep-Learning-Derived Evaluation Metrics Enable Effective Benchmarking of Computational Tools for Phosphopeptide Identification.
Deep-Learning-Derived Evaluation Metrics Enable Effective Benchmarking of Computational Tools for Phosphopeptide Identification.
复制标题
DOI：
10.1016/j.mcpro.2021.100171
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Zhang B
中科院分区：
文献类型：
--
作者：
Jiang W;Wen B;Li K;Zeng WF;da Veiga Leprevost F;Moon J;Petyuk VA;Edwards NJ;Liu T;Nesvizhskii AI;Zhang B
基于串联质谱 (MS/MS) 的磷酸化蛋白质组学是一种用于全局磷酸化分析的强大技术。然而，将四个计算流程应用于人类癌症研究中典型的基于质谱 (MS) 的磷酸化蛋白质组数据集时，我们观察到报告的磷酸肽鉴定和磷酸位点定位结果之间存在很大差异，这强调了基准测试的迫切需要。虽然人们已经努力使用合成磷酸肽的数据来比较计算管道的性能，但由于缺乏适当的评估指标，涉及实际应用数据的评估在很大程度上仅限于比较磷酸肽鉴定的数量。我们研究了三个深度学习衍生的特征作为潜在的评估指标：磷酸位点概率、Delta RT 和光谱相似性。预测的磷酸位点概率由 MusiteDeep 计算，它提供了先前报道的高精度； Delta RT 定义为 AutoRT 观察到的 RT 与预测的 RT 之间的绝对保留时间 (RT) 差异；光谱相似度定义为 pDeep2 观测到的光谱与预测的光谱之间的皮尔逊相关系数。使用合成肽数据集，我们发现当不正确的 PSM 涉及错误的肽序列时，甚至当不正确的 PSM 仅由不正确的磷酸位点定位引起时，Delta RT 和光谱相似性都可以很好地区分正确和不正确的肽谱匹配 (PSM)。基于这些结果，我们使用所有三个深度学习衍生的特征作为评估指标来比较不同磷酸蛋白质组数据集上的不同计算管道，并展示它们在管道性能基准测试中的实用性。本研究中演示的基准指标将使用户能够选择用于磷酸蛋白质组数据的常规分析的计算管道和参数，并将为开发人员改进计算方法提供指导。计算方法的选择很大程度上影响磷酸肽的鉴定。深度学习衍生的指标可以有效地区分正确和错误的 PSM。新颖的指标可以对实际应用数据进行计算方法比较。基于串联质谱 (MS/MS) 的磷酸化蛋白质组学是一种用于全局磷酸化分析的强大技术。然而，对同一数据集应用不同的计算流程可能会产生截然不同的磷酸肽识别结果，这凸显了基准测试的迫切需要。我们提出了三个深度学习衍生的基准指标。本研究中演示的基准指标将使用户能够选择用于磷酸蛋白质组数据的常规分析的计算管道和参数，并将为开发人员改进计算方法提供指导。
Tandem mass spectrometry (MS/MS)-based phosphoproteomics is a powerful technology for global phosphorylation analysis. However, applying four computational pipelines to a typical mass spectrometry (MS)-based phosphoproteomic dataset from a human cancer study, we observed a large discrepancy among the reported phosphopeptide identification and phosphosite localization results, underscoring a critical need for benchmarking. While efforts have been made to compare performance of computational pipelines using data from synthetic phosphopeptides, evaluations involving real application data have been largely limited to comparing the numbers of phosphopeptide identifications due to the lack of appropriate evaluation metrics. We investigated three deep-learning-derived features as potential evaluation metrics: phosphosite probability, Delta RT, and spectral similarity. Predicted phosphosite probability is computed by MusiteDeep, which provides high accuracy as previously reported; Delta RT is defined as the absolute retention time (RT) difference between RTs observed and predicted by AutoRT; and spectral similarity is defined as the Pearson’s correlation coefficient between spectra observed and predicted by pDeep2. Using a synthetic peptide dataset, we found that both Delta RT and spectral similarity provided excellent discrimination between correct and incorrect peptide-spectrum matches (PSMs) both when incorrect PSMs involved wrong peptide sequences and even when incorrect PSMs were caused by only incorrect phosphosite localization. Based on these results, we used all the three deep-learning-derived features as evaluation metrics to compare different computational pipelines on diverse set of phosphoproteomic datasets and showed their utility in benchmarking performance of the pipelines. The benchmark metrics demonstrated in this study will enable users to select computational pipelines and parameters for routine analysis of phosphoproteomics data and will offer guidance for developers to improve computational methods. Computational method selection substantially affects phosphopeptide identification. Deep-learning-derived metrics effectively discriminate correct and incorrect PSMs. Novel metrics enable computational method comparison on real application data. Tandem mass spectrometry (MS/MS)-based phosphoproteomics is a powerful technology for global phosphorylation analysis. However, applying different computational pipelines to the same dataset may produce substantially different phosphopeptide identification results, underscoring a critical need for benchmarking. We present three deep-learning-derived benchmark metrics. The benchmark metrics demonstrated in this study will enable users to select computational pipelines and parameters for routine analysis of phosphoproteomics data and will offer guidance for developers to improve computational methods.