EAGER: Efficient Algorithms for Dimensionality Reduction and Clustering Using Disk-Based Matrices

EAGER：基于磁盘矩阵的高效降维和聚类算法大数据集的线性高斯模型计算的特点是矩阵操作重，迭代方法收敛慢。考虑到大型数据集是从磁盘存储和检索的，并且在数据库系统中模型也在磁盘上操作，效率问题变得更糟。尽管它们很重要，但很少有研究工作试图利用数据库系统技术来适应这个大家庭的高斯模型。本提案研究如何改进线性高斯模型的算法，以分析大型高维数据集，在次要存储（即磁盘）上操作矩阵，使用少量主存储（即RAM内存）。本文研究的模型包括用于降维的最大似然因子分析和用于聚类的混合高斯分布。这项建议的教育部分涉及两项主要活动。第一项活动是制定一项计划，让弱势和少数族裔高中生接触数据挖掘研究和实践，以鼓励他们学习计算机科学。第二项活动涉及加强休斯敦大学目前关于数据挖掘的研究和教学。本研究项目需要发现通用算法原理，为一系列统计模型执行增量矩阵计算，了解如何总结大型数据集，保留多个模型所需的统计属性，并提出针对这些模型量身定制的新数据库技术，能够在二级存储上执行有效的矩阵操作。增量计算很难实现，因为线性高斯模型的方法需要对整个数据集进行迭代。摘要需要变换复杂的矩阵方程，考虑高维数、大数据集和数值稳定性，同时保持模型精度。开发结合主存储和辅助存储的矩阵优化与优化仅在主存储上工作的矩阵算法有很大不同。这项研究工作需要数学知识来推广、优化和转换线性高斯模型的计算。另一方面，它需要数据库系统的专业知识，了解如何在二级存储上组织和索引不同的矩阵，以实现高效的读写。更广泛的影响该建议将对大型、复杂、高维科学数据集的分析产生广泛的影响，并通过增量模型计算能力增强数据库系统。我们计划在科学数据集上应用和测试我们提出的算法和技术，包括地理、医学和生物数据集等。

EAGER: Efficient Algorithms for Dimensionality Reduction and Clustering Using Disk-Based Matrices Carlos Ordonez1. Research and Education Proposal Linear Gaussian models on large data sets computation is characterized by heavy matrix manipulation and iterative methods with slow convergence. Efficiency issues become worse considering the fact that large data sets are stored and retrieved from disk and that in a database system models are manipulated on disk as well. Despite their importance there is scarce research work that attempts to adapt this big family of Gaussian models exploiting database systems techniques. This proposal studies how to improve algorithms for linear Gaussian models to analyze large, high dimensional, data sets, manipulating matrices on secondary storage (i.e. disk), using a small amount of primary storage (i.e. RAM memory). The models studied herein include maximum likelihood factor analysis for dimensionality reduction and mixtures of Gaussian distributions to perform clustering.The educational component of this proposal involves two main activities. The first activity is to develop a plan to expose disadvantaged and minority high school students to data mining research and practice in order to encourage them to study computer science. The second activity involves enhancing current research and teaching of data mining at the University of Houston.2. Intellectual MeritThis research project requires the discovery of common algorithmic principles to perform incremental matrix computations for a family of statistical models, understanding how to summarize large data sets, preserving their statistical properties required by multiple models and proposing new database techniques tailored for such models, capable of performing efficient matrix manipulation on secondary storage. Incremental computations are difficult to attain because methods for linear Gaussian models require iterations on the entire data set. Summarization requires transforming complex matrix equations considering high dimensionality, large data set size and numerical stability, preserving model accuracy. Developing matrix optimizations combining primary and secondary storage is quite different from optimizing a matrix algorithm that works only on primary storage.This research work requires mathematical knowledge to generalize, optimize and transform the computation of linear Gaussian models. On the other hand, it needs database systems expertise on how to organize and index diverse matrices on secondary storage for efficient reading and writing.3. Broader ImpactThis proposal will have a broad impact on the analysis of large, complex, high dimensional scientific data sets and enhancing database systems with incremental model computation capabilities. We plan to apply and test our proposed algorithms and techniques on scientific data sets, including geographical, medical and biological data sets, among others.