(Re)designing Clustering Algorithms for Big Data

我们目前的数据生成速度与存储容量的增加相结合，催生了计算领域的新范式。例如，根据IBM网站的数据，Facebook上每分钟大约有69.5万条状态更新和1100万条即时消息发送。然而，许多组织都面临着拥有大量数据但对它们知之甚少的问题。集群方法帮助自动识别一组数据对象的未观察到的组，并且当前正在通过可用数据的大小、种类和性质，即所谓的“大数据”，从根本上改变。本研究计划致力于开发适用于大数据集群的可扩展算法。这将通过以下两个途径实现：(I)利用它们的主要理论思想，重新设计众所周知的成功的系列算法以实现可伸缩性；(Ii)使用与大数据相关的新编程范例开发新的算法和启发式算法。 总而言之，这项研究计划的目标是： 答：对现有的大数据集群方法进行广泛的调查，以便提供关于哪些方法可以适应大数据的完整全景。 B.重新设计成功的序列聚类算法，利用它们的主要理论思想与来自大数据的新编程范例和计算工具一起工作。 C.开发半监督大数据聚类算法，将用户提供的补充信息纳入到聚类决策过程中。 D.为新出现的应用程序开发大数据集群算法。 E.以有保证的有效性提供大数据软件和集群算法的存储库。 在这项研究计划中开发的软件和算法预计将构成该领域的新基准，使更大的数据集能够高效和有效地处理，从而在商业、工业和学术界产生新的见解。此外，鉴于加拿大和全球大数据专家短缺，该研究计划将有助于形成和培训能够掌握大数据爆炸所产生的科学和技术问题的专家。 从长远来看，该研究计划的结果将支持物联网(IoT)时代的数据挖掘驱动的决策制定，在物联网时代，从最多样化的来源和设备(例如车辆、传感器、家用电器等)实时生成海量数据。通过结合海量数据处理、机器学习技术和算法，物联网设备将能够大规模执行集群和其他分类任务。

Our current speed of data generation combined with storage capacity increases have given rise to new paradigms in computing. For example, according to the IBM's website, there are approximately 695,000 status updates and 11 million instant messages sent every minute on Facebook. However, many organizations have faced the problem of having a lot of data, but poor knowledge about them. Clustering methods help to automatically identify unobserved groups for a set of data objects, and are currently being radically transformed by the size, the variety and the nature of the available data, i.e., by so-called “Big Data”. This research program focuses on the development of scalable algorithms for Big data clustering. This will be achieved both by: (i) redesigning well-known successful serial algorithms for scalability, leveraging their main theoretical ideas; and (ii) developing new algorithms and heuristics using new programming paradigms associated with Big Data. In summary, the objectives of this research program are : A. Produce an extensive survey of the existing Big Data clustering methods in order to provide a complete panorama about which ones can be adapted to approach Big data. B. Redesign successful serial clustering algorithms, leveraging their main theoretical ideas to work with the new programming paradigms and computational tools from Big Data. C. Develop algorithms for semi-supervised Big Data clustering, incorporating supplementary information provided by the user into the clustering decision process. D. Develop Big Data clustering algorithms for new emerging applications. E. Provide a repository of Big Data software and clustering algorithms with guaranteed effectiveness. The software and algorithms developed in this research program are expected to constitute new benchmarks to the field, allowing larger datasets to be tackled with efficiency and effectiveness, leading to new insights in commerce, industry and academia. Moreover, given the Big Data expert shortage in Canada and worldwide, this research program will help to form and train specialists who will be able to master the scientific and technological issues emerging from the Big Data explosion. In the long term, the findings of this research program will support data mining-driven decision making in the Internet of Things (IoT) era in which huge amounts of data are generated in real-time from the most varied sources and devices (e.g. vehicles, sensors, home appliances, etc.). By combining massive data processing, machine learning techniques and algorithms, IoT devices will be able to perform clustering as well as other classification tasks on a large scale.