SHF: Medium: Collaborative Research: Scalable Algorithms for Spatio-temporal Data Analysis

随着望远镜、对撞机、传感器和设备等大型仪器的发展和部署，超级计算机计算能力的加速提出了一个基本问题。“洞察力和知识发现的时间能以同样的指数速度减少吗？”目前的答案显然是“不”，因为从海量数据集中结合分析、挖掘和发现知识的关键步骤远远落后于软件、模拟和数据生成方面的进步。数据分析需要“数据驱动”的计算和分析。这需要可伸缩的软件来进行数据简化、近似、分析、统计和自下而上的发现。为了在科学发现方面取得重大飞跃，需要处理大量数据的可扩展并行分析软件。本项目开发创新的、可扩展的、可持续的数据分析算法，以实现在高性能并行计算机上对海量数据的分析和挖掘，包括：(1)适用于时空数据、海量图分析、社区计算以及时变图、不同类型数据和不同数据特征的模式检测的自下而上和无监督数据聚类算法；(2)时空数据变化检测与异常检测；(3)在一定的时间和空间约束下跟踪移动数据和集群动态。这些并行算法使用科学应用产生的大量数据，如天体物理学、宇宙学模拟、气候建模和社会网络分析，在现代高性能并行计算机上进行结果验证和性能评估。该项目通过大数据并行分析软件直接解决了对时空数据分析、性能可扩展性和大规模科学发现编程生产力的关键需求。这项工作将通过加速科学发现，影响巨大的社会效益和科学重要性的应用，如气候理解、环境可持续性、天体物理学、生物学和医学。此外，开发的软件基础结构可以在商业应用程序中使用和采用，例如商业、社会、安全、药物发现等。源代码向公众开放，供所有社区进行调整、构建、定制和贡献，从而增加其价值和用途。

Acceleration of computing power of supercomputers along with development and deployment of large instruments such as telescopes, colliders, sensors and devices raises one fundamental question. "Can the time to insight and knowledge discovery be reduced at the same exponential rate?" The answer currently is clearly "NO", because a critical step that combines analytics, mining and discovering knowledge from the massive datasets has lagged far behind advances in software, simulation and generation of data. Analysis of data requires "data-driven" computing and analytics. This entails scalable software for data reduction, approximations, analysis, statistics, and bottom-up discovery. Scalable and parallel analytics software for processing large amount of data is required in order to make a significant leap forward in scientific discoveries. This project develops innovative, scalable, and sustainable data analytics algorithms to enable analysis and mining of massive data on high-performance parallel computers, which include (1) bottom-up and unsupervised data clustering algorithms that are suitable for spatio-temporal data, massive graph analytics, community computations, and detection of patterns in time-varying graphs, different types of data, and different data characteristics; (2) change detection and anomaly detection in spatio-temporal data; and (3) tracking moving data and cluster dynamics within certain time and space constraints. These parallel algorithms use the massive amount of data generated from scientific applications, such as astrophysics, cosmology simulations, climate modeling, and social networking analysis, for result verification and performance evaluation on modern high-performance parallel computers.This project directly addresses the critical needs for spatio-temporal data analysis, performance scalability, and programming productivity of large-scale scientific discovery via parallel analytics software for big data. This work will impact applications of enormous societal benefits and scientific importance such as climate understanding, environmental sustainability, astrophysics, biology and medicine by accelerating scientific discoveries. Furthermore, the developed software infrastructure can be used and adopted in commercial applications, such as commerce, social, security, drug discovery, and so on. The source codes are open to the public for all community to adapt, build-upon, customize and contribute to, thereby multiplying its value and usage.