(Re)designing Clustering Algorithms for Big Data

我们目前的数据生成速度与存储容量的增加相结合，产生了新的计算模式。例如，根据IBM的网站，Facebook上每分钟大约有695，000条状态更新和1100万条即时消息发送。然而，许多组织都面临着拥有大量数据，但对它们缺乏了解的问题。聚类方法有助于为一组数据对象自动识别未观察到的组，并且目前正在根据可用数据的大小、种类和性质进行根本性的转变，即，所谓的“大数据”。该研究计划专注于大数据聚类的可扩展算法的开发。这将通过以下两种方式实现：（i）重新设计知名的成功串行算法以实现可扩展性，利用其主要理论思想;（ii）使用与大数据相关的新编程范式开发新算法和算法。 * 总的来说，本研究计划的目标是：*A。对现有的大数据聚类方法进行广泛的调查，以提供关于哪些方法可以适用于大数据的完整全景。*B。重新设计成功的串行聚类算法，利用它们的主要理论思想与来自大数据的新编程范式和计算工具一起工作。*C.开发半监督大数据聚类算法，将用户提供的补充信息纳入聚类决策过程。*D.为新兴应用开发大数据聚类算法。*E.提供大数据软件和聚类算法的存储库，保证有效性。*该研究项目开发的软件和算法有望成为该领域的新基准，使更大的数据集能够高效、有效地处理，从而为商业、工业和学术界带来新的见解。此外，鉴于加拿大和全球大数据专家短缺，该研究计划将有助于形成和培养能够掌握大数据爆炸中出现的科学和技术问题的专家。从长远来看，这项研究计划的结果将支持物联网（IoT）时代的数据挖掘驱动的决策，在物联网时代，大量数据从各种来源和设备（例如车辆，传感器，家用电器等）实时生成。通过结合大量数据处理、机器学习技术和算法，物联网设备将能够大规模执行聚类以及其他分类任务。

Our current speed of data generation combined with storage capacity increases have given rise to new paradigms in computing. For example, according to the IBM's website, there are approximately 695,000 status updates and 11 million instant messages sent every minute on Facebook. However, many organizations have faced the problem of having a lot of data, but poor knowledge about them. Clustering methods help to automatically identify unobserved groups for a set of data objects, and are currently being radically transformed by the size, the variety and the nature of the available data, i.e., by so-called “Big Data”. This research program focuses on the development of scalable algorithms for Big data clustering. This will be achieved both by: (i) redesigning well-known successful serial algorithms for scalability, leveraging their main theoretical ideas; and (ii) developing new algorithms and heuristics using new programming paradigms associated with Big Data. ***In summary, the objectives of this research program are :***A. Produce an extensive survey of the existing Big Data clustering methods in order to provide a complete panorama about which ones can be adapted to approach Big data. ***B. Redesign successful serial clustering algorithms, leveraging their main theoretical ideas to work with the new programming paradigms and computational tools from Big Data. ***C. Develop algorithms for semi-supervised Big Data clustering, incorporating supplementary information provided by the user into the clustering decision process. ***D. Develop Big Data clustering algorithms for new emerging applications. ***E. Provide a repository of Big Data software and clustering algorithms with guaranteed effectiveness.***The software and algorithms developed in this research program are expected to constitute new benchmarks to the field, allowing larger datasets to be tackled with efficiency and effectiveness, leading to new insights in commerce, industry and academia. Moreover, given the Big Data expert shortage in Canada and worldwide, this research program will help to form and train specialists who will be able to master the scientific and technological issues emerging from the Big Data explosion.***In the long term, the findings of this research program will support data mining-driven decision making in the Internet of Things (IoT) era in which huge amounts of data are generated in real-time from the most varied sources and devices (e.g. vehicles, sensors, home appliances, etc.). By combining massive data processing, machine learning techniques and algorithms, IoT devices will be able to perform clustering as well as other classification tasks on a large scale.**