On the statistical foundation of a recent single molecule FRET benchmark.
On the statistical foundation of a recent single molecule FRET benchmark.
复制标题
基于最近单分子 FRET 基准的统计基础。
DOI：
10.1038/s41467-024-47733-3
复制
发表时间：
2024
影响因子：
16.6
通讯作者：
Pressé,Steve
中科院分区：
文献类型：
--
作者：
Saurabh,Ayush;Xu,LanceWQ;Pressé,Steve
最近发表在《自然通讯》上的一项基准研究，题为“从单分子Förster共振能量转移（FRET）轨迹推断动力学速率常数的分析工具的盲基准”，参考文献1比较了多种FRET分析工具。为了对这些工具进行基准测试，组织者向FRET工具开发人员提供了综合训练数据集，以及数据生成的Matlab脚本。这些团队将自己的工具应用于提供的挑战数据集。然后，组织者使用包括变异系数在内的指标比较了所有工具的输出（例如从提供的数据中学习到的动力学参数）。该工具还应用于实验数据。在这里，我们证明基准有利于FRET工具，使其与简化数据生成过程中存在的假设相似（即，高斯模型很好地分析高斯噪声数据），并且，出于同样的原因，在包含FRET物理特征的工具获得的参数估计中导致偏差和不正确的不确定性，否则这些不被纳入用于测试FRET分析工具性能的生成数据中。例如，实际生成的用于测试FRET分析工具性能的数据应该包含常见的物理来源的FRET噪声源，以便正确地基准测试工具及其在信噪比（SNR）制度中的鲁棒性。举例来说，这些源包括光子散点噪声、探测器噪声和光谱串扰。相反，基准组织者生成的数据缺乏这些特征，而是在高斯噪声假设下生成的。在这里，我们通过测试在大多数基准测试（hide - markury 3和MASH-FRET 4,5）中表现良好的参与工具，对我们仅用两个现实和广泛的特征（泊松射击噪声和检测器串扰）模拟的数据进行测试，来评估FRET分析工具在排除实际FRET特征的数据上的性能的后果。在此过程中，我们证明：(1)依赖高斯噪声模型的基准参与者在测试高斯噪声数据（如为基准生成的数据）时可预测地表现良好（图1），因此，在分析中添加物理特征（如泊松噪声）会导致不准确的参数和不确定性估计，因为它会在用于分析的模型之间产生不匹配
A benchmark recently published in Nature Communications entitled “A blind benchmark of analysis tools to infer kinetic rate constants from single-molecule Förster Resonance Energy Transfer (FRET) trajectories” by ref. 1 compared multiple FRET analysis tools. To benchmark these tools, the organizers provided FRET tool developers synthetic training datasets, along with the data-generating Matlab script. The teams applied their own tools to the challenge datasets provided. The organizers then compared the output (such as learned kinetic parameters from the supplied data) for all tools using metrics including coefficients of variation. The tools were also applied to experimental data.Here we demonstrate that the benchmark favors FRET tools making similar assumptions to those present in the simplified data generation process (ie, Gaussian models analyze Gaussian noise data well) and, by the same token, leads to bias and incorrect uncertainty in parameter estimates obtained by tools incorporating physical features of FRET that are otherwise not incorporated into the data generated used to test the performance of FRET analysis tools. As an example, realistically generated data used to test the performance of FRET analysis tools should incorporate common FRET noise sources of physical origin in order to properly benchmark tools and their robustness across signal-to-noise ratio (SNR) regimes. These sources include, as examples, photon shot noise, detector noise, and spectral crosstalk 2 just to name a few. Instead, the data generated by the benchmark organizers lacked these features and were instead generated under Gaussian noise assumptions. Here we investigate the consequences of evaluating the performance of FRET analysis tools on data excluding realistic FRET features by testing on participating tools that performed well in most of the benchmark tests, Hidden-Markury 3 and MASH-FRET 4, 5, on data we simulate with just two realistic and widespread features: Poisson shot noise and detector crosstalk. In doing so, we demonstrate that:(1) benchmark participants, relying on a Gaussian noise model, predictably do well when tested on Gaussian noise data like the data generated for the benchmark (Fig. 1) and, as a consequence, adding physical features to one’s analysis, such as Poissonian noise, results in inaccurate parameter and uncertainty estimates as it creates a mismatch between the model used to analyze