SPX: Collaborative Research: Multicore to Wide Area Analytics on Streaming Data

SPX：协作研究：流数据的多核到广域分析

• 批准号: 1725702
• 负责人: Goce Trajcevski
• 金额: $ 30.8万
• 依托单位: Iowa State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-15 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1725702&HistoricalAwards=false
• 关键词: SPX; Collaborative; Research; Multicore; Wide

In today's big data era, there is an urgent need for methods that can quickly derive analytical insights from large volumes of data that are continuously generated. Such streaming data include video, audio, activity logs, and sensor data, and are generated on a massive scale all over the world. The need for real-time streaming analytics can only be fulfilled with the help of appropriately designed parallel and distributed algorithms. However, parallel and distributed computing systems come in a variety of shapes and sizes, and algorithms should be designed to match the characteristics of the underlying system. This project develops methods for analyzing massive streaming data on computing systems ranging from machines with multiple cores sharing memory to geo-distributed data centers communicating over wide-area networks. The results of this research are expected to improve the efficiency, latency, and throughput of streaming analytics. Due to the foundational nature of the analytical tasks considered, results of this project will impact disciplines that use large-scale machine learning and graph analytics, including cybersecurity, social network analysis, and transportation. Resulting software will be released as toolkits on stream processing platforms, and deployed in a smart-city camera infrastructure. Synergy between the research goals and the teaching goals of the PIs will lead to new instructional material in existing courses as well as development of new courses in data analytics. Individuals from underrepresented groups will be included as a part of the project. The project will benefit from and strengthen collaborations between academia, industry, and national labs on streaming analytics. The first technical thrust of the project is on designing shared memory parallel algorithms for computation on data streams, that can achieve a high throughput and fast convergence for complex analytics tasks. The second thrust is on designing distributed streaming algorithms that can tolerate variable communication delays and adapt to available bandwidth in a wide-area network, through identifying good tradeoffs between freshness of results and volume of communication. These advances will be studied in the context of fundamental graph analytics and machine learning tasks such as subgraph counting, graph connectivity and clustering, matrix factorization, and deep networks. The project will utilize the vast body of theory and techniques developed in the realm of parallel computing in the design of methods for processing streaming data, leading to a toolkit of techniques that can be reused across applications. The project will also lead to advances in sequential streaming and incremental algorithms for certain problems; for instance, problems in machine learning that use iterative convergent methods. Based on the techniques designed, the project will design and build a hierarchical parameter server that operates effectively across the spectrum from multicore machines to data centers to wide-area data sources.

在当今的大数据时代，迫切需要能够从不断生成的大量数据中快速获得分析见解的方法。 这种流数据包括视频、音频、活动日志和传感器数据，并且在世界各地大规模生成。 实时流媒体分析的需求只能在适当设计的并行和分布式算法的帮助下才能实现。然而，并行和分布式计算系统有各种各样的形状和大小，算法的设计应该与底层系统的特性相匹配。该项目开发了用于分析计算系统上的大量流数据的方法，这些计算系统包括从具有多核共享内存的机器到通过广域网通信的地理分布数据中心。这项研究的结果有望提高流媒体分析的效率、延迟和吞吐量。 由于所考虑的分析任务的基础性质，该项目的结果将影响使用大规模机器学习和图形分析的学科，包括网络安全，社交网络分析和交通。由此产生的软件将作为流处理平台上的工具包发布，并部署在智能城市摄像机基础设施中。PI的研究目标和教学目标之间的协同作用将导致现有课程中的新教材以及数据分析新课程的开发。来自代表性不足群体的个人将被纳入该项目。该项目将受益于并加强学术界，工业界和国家实验室之间在流媒体分析方面的合作。该项目的第一个技术重点是设计用于数据流计算的共享内存并行算法，可以实现复杂分析任务的高吞吐量和快速收敛。第二个推力是设计分布式流算法，可以容忍可变的通信延迟，并适应在广域网中的可用带宽，通过确定结果的新鲜度和通信量之间的良好权衡。这些进展将在基础图分析和机器学习任务的背景下进行研究，例如子图计数，图连接和聚类，矩阵分解和深度网络。该项目将利用并行计算领域开发的大量理论和技术来设计处理流数据的方法，从而形成一个可在应用程序中重复使用的技术工具包。该项目还将导致某些问题的顺序流和增量算法的进步;例如，使用迭代收敛方法的机器学习问题。基于所设计的技术，该项目将设计和构建一个分层参数服务器，该服务器可以在从多核机器到数据中心再到广域数据源的范围内有效运行。

项目成果

Incremental Maintenance of Maximal Bicliques in a Dynamic Bipartite Graph

• DOI: 10.1109/tmscs.2018.2802920
• 发表时间: 2018-07
• 期刊: IEEE Transactions on Multi-Scale Computing Systems
• 作者: A. Das;Srikanta Tirthapura

Shared-Memory Parallel Maximal Biclique Enumeration

• DOI: 10.1109/hipc.2019.00016
• 发表时间: 2018-12
• 期刊: 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC)
• 作者: A. Das;Srikanta Tirthapura

Weighted Reservoir Sampling from Distributed Streams

从分布式流中进行加权水库采样

• DOI: 10.1145/3294052.3319696
• 发表时间: 2019
• 期刊: Proceedings of the 38th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems
• 作者: Jayaram, Rajesh;Sharma, Gokarna;Tirthapura, Srikanta;Woodruff, David P.
• 通讯作者: Woodruff, David P.

Stratified random sampling from streaming and stored data

• DOI: 10.1007/s10619-020-07315-w
• 发表时间: 2020-10-23
• 期刊: DISTRIBUTED AND PARALLEL DATABASES
• 影响因子: 1.2
• 作者: Nguyen, Trong Duc;Shih, Ming-Hung;Xu, Bojian
• 通讯作者: Xu, Bojian

Learning Graphical Models from a Distributed Stream

从分布式流中学习图形模型

• DOI: 10.1109/icde.2018.00071
• 发表时间: 2018
• 期刊: Proceedings of the IEEE 34th International Conference on Data Engineering (ICDE
• 作者: Zhang, Yu;Tirthapura, Srikanta;Cormode, Graham
• 通讯作者: Cormode, Graham