Out-of-core singular value decomposition
Out-of-core singular value decomposition
复制标题
核外奇异值分解
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Stefan Bordag
中科院分区：
文献类型：
--
作者：
V. Demchik;M. Bacák;Stefan Bordag
奇异值分解（SVD）是一种标准的矩阵分解技术，可产生矩阵的最佳低秩逼近。它具有多种应用，包括机器学习，数据科学和信号处理。然而，许多常见的问题涉及到非常大的矩阵，无法容纳在商品计算机的主存储器中，使得使用标准的SVD算法不切实际，该算法假设快速随机访问或大量空间用于中间计算。为了解决这个问题，我们已经实现了一个核外（外部存储器）的随机SVD解决方案，是完全可扩展的，有效的并行化。该解决方案在任意小的内存限制内分解任意大大小的密集和稀疏矩阵，并根据需要有效地使用核外存储。它使用了一种创新的矩阵分区技术，适合于核外和并行处理，以及内存和I/O使用规划，自动负载平衡，性能调优，并使许多其他实际增强成为可能，以当前的最先进的。（通常是HDD或SSD），用户可以恢复中断的操作，而不必重新计算先前执行的步骤，解决了分解非常大的矩阵的主要实际问题。
Singular value decomposition (SVD) is a standard matrix factorization technique that produces optimal low-rank approximations of matrices. It has diverse applications, including machine learning, data science and signal processing. However, many common problems involve very large matrices that cannot fit in the main memory of commodity computers, making it impractical to use standard SVD algorithms that assume fast random access or large amounts of space for intermediate calculations. To address this issue, we have implemented an out-of-core (external memory) randomized SVD solution that is fully scalable and efficiently parallelizable. This solution factors both dense and sparse matrices of arbitrarily large size within arbitrarily small memory limits, efficiently using out-of-core storage as needed. It uses an innovative technique for partitioning matrices that lends itself to out-of-core and parallel processing, as well as memory and I/O use planning, automatic load balancing, performance tuning, and makes possible a number of other practical enhancements to the current state-of-the-art. Furthermore, by using persistent external storage (generally HDDs or SSDs), users can resume interrupted operations without having to recalculate previously performed steps, solving a major practical problem in factoring very large matrices.
DOI：
--
发表时间：
2012
期刊：
--
影响因子：
--
作者：
P. Martinsson;N. Halko
通讯作者：
P. Martinsson;N. Halko