EFFICIENT SPECTRAL APPROACHES FOR FINDING UNDERLYING STRUCTURES IN BIG DATA

 描述(申请人提供)L最近，我们开发了几种光谱方法，用于分析属于大数据(BD)类别的超大型基因组数据集或完整数据库。第一种方法是基于随机化来执行奇异值分解或主成分分析，相对于所有通用软件包中实现的标准Lanczos算法，这种随机化方法可以显著加快其特征向量和特征值的计算。更有效地计算主成分分析和奇异值分解，可能会给基于主成分分析和奇异值分解的无数生物医学应用带来革命性的变化，例如在非常大的地理信息系统中进行人口分层。这些算法比经典(确定性)方法产生更高的精度，能够处理太大而无法存储的数据流，并且很容易并行化以用于多核微处理器。我们的第二种新方法是一种无监督谱学习方法。它提供了惊人的概念简单性的新的数学见解，用于在不访问验证数据的情况下对多个竞争算法进行排序，并用于在缺乏基本事实的情况下以最佳方式组合该算法集合以获得改进的预测。构建一种工具，为终端用户提供对用于分析基因组数据的算法进行最佳排序或组合的选项，是消除终端用户或生物信息学家的困惑的实用而有效的解决方案，因为由不同工具推断的大量生物学结果经常是不一致的，这些最终用户或生物信息学家面临着为他们的研究选择哪个工具的需要。选择性能最好的算法或流水线是至关重要的，因为它通常可以导致从大规模并行测序实验中读出的质量的显著提高。此外，组合这些工具通常会产生优于最佳性能算法的性能。我们的目标是建立一个团队，其重点是提供和传播光谱BD工具和方法的全面实施，这些工具和方法在生物医学科学、临床研究和医疗保健提供的范围内具有广泛的适用性。具体地说，我们将为基因组学和生物医学应用开发可扩展的PCA和SVD，进一步改进我们用于对竞争管道的性能进行排名的频谱方法，并将它们结合在一起，在不访问验证数据的情况下实现更好的预测。此外，我们将开发可扩展的降维技术，用于组织生物医学应用中的BD。

 DESCRIPTION (provided by applicant)l Recently we developed several spectral approaches for analyzing very large genomics datasets or complete databases that fall into the category of big data (BD). The first approach is designed to perform SVD or PCA based on randomization that can dramatically accelerate the computation of their eigenvectors and eigenvalues relative to the standard Lanczos algorithm implemented in all common software packages. Computing PCA and the SVD more efficiently could revolutionize the innumerable biomedical applications based on PCA and the SVD, e.g. population stratification in very large GWAS. These algorithms produce higher accuracy than classical (deterministic) methods, enable the processing of data streams that are too large to store, and parallelize easily to be used in multicore microprocessors. Our second novel approach is an unsupervised spectral learning method. It provides new mathematical insights of striking conceptual simplicity for ranking multiple competing algorithms without access to validation data and for optimally combining this ensemble of algorithms to obtain improved predictions in the absence of ground truth. Constructing a tool that provides end users an option to optimally rank or combine algorithms for analysis of genomics data is a practical and efficient solution to remove the confusion among end-users or bioinformaticians who are faced with the need to decide which tool to choose for their study, as a large number of biological results inferred by the different tools are often in disagreement. The choice of the best performing algorithm or pipeline is essential as it can often lead to substantial improvement in quality of the readout from massively parallel sequencing experiments. Moreover, combining these tools typically results in performance superior to the best performing algorithm. Our goal is to establish a team whose focus is to provide and disseminate full-blown implementations of spectral BD tools and methods that have broad applicability across the spectrum of biomedical sciences, clinical research, and healthcare delivery. Specifically we will develop scalable PCA and SVD for Genomics and biomedical applications, further advance our spectral method for ranking the performance of competing pipelines and combine them to achieve better predictions without access to validation data. Moreover, we will develop scalable dimensional reduction techniques for organizing BD from biomedical applications.