SI2-SSE: Scalable Big Data Clustering by Random Projection Hashing

本计画发展一个分散式演算法，以安全地分群高维资料集。 健康和生物学领域从数据聚类可扩展性中受益匪浅。 生物信息学问题，如微阵列聚类，蛋白质-蛋白质相互作用聚类，医疗资源决策，医学图像处理和流行病学事件的聚类都有助于从更大的数据集大小中受益。 正在开发的算法称为随机投影哈希或RPHash，利用局部敏感哈希（LSH）和多探针随机投影的计算可扩展性和并行速度的线性可实现增益。 此外，RPHash通过对数据的破坏性操作提供数据匿名化，防止超出标准最佳实践数据库安全方法的去匿名化攻击。 RPHash将可部署在运行MapReduce的Hadoop（MRv 2）实现的商用云资源上。 通用云处理解决方案的开发使研究人员能够使用几乎无限的商业处理资源来扩展他们的处理需求。RPHash算法使用了数据挖掘中的各种最新技术，沿着采用了一种新的方法来实现分布式系统上的算法可扩展性。 RPHash的基本直觉是将联合收割机多探针随机投影与离散空间量化相结合。 高密度的区域，然后被视为质心候选人。 为了遵循常见的参数化k-means方法，将选择前k个区域。 对随机的，因此非确定性的，聚类算法的重点是在计算中有点不常见，但常见的不适定，组合限制的问题，如聚类和分区。 尽管理论结果表明，k-均值具有指数最坏情况下的复杂度，许多真实的世界的问题往往公平的k-means和其他类似的算法下更好。

This project plans to develop a distributed algorithm for secure clustering of high dimensional data sets. Fields in health and biology are significantly benefited by data clustering scalability. Bioinformatic problems such as Micro Array clustering, Protein-Protein interaction clustering, medical resource decision making, medical image processing, and clustering of epidemiological events all serve to benefit from larger dataset sizes. The algorithm under development, called Random Projection Hash or RPHash, utilizes aspects of locality sensitive hashing (LSH) and multi-probe random projection for computational scalability and linear achievable gains from parallel speed. Furthermore, RPHash provides data anonymization through destructive manipulation of the data preventing de-anonymization attacks beyond standard best practices database security methods. RPHash will be deployable on commercially available cloud resources running the Hadoop (MRv2) implementation of MapReduce. The exploitation of general purpose cloud processing solutions allows researchers to scale their processing needs using virtually limitless commercial processing resources.The RPHash algorithm uses various recent techniques in data mining along with a new approach toward achieving algorithmic scalability on distributed systems. The basic intuition of RPHash is to combine multi-probe random projection with discrete space quantization. Regions of high density are then regarded as centroid candidates. To follow common parameterized, k-means methods, the top k regions will be selected. The focus on a randomized, and thus non-deterministic, clustering algorithm is somewhat uncommon in computing, but common for ill-posed, combinatorially restrictive problems such as clustering and partitioning. Despite theoretical results showing that k-means has an exponential worst case complexity, many real world problems tend to fair much better under k-means and other similar algorithms.