SPX: Collaborative Research: Multicore to Wide Area Analytics on Streaming Data

在当今的大数据时代，迫切需要能够从持续生成的海量数据中快速获得分析见解的方法。这类流数据包括视频、音频、活动日志和传感器数据，并在世界各地大规模生成。只有在适当设计的并行和分布式算法的帮助下，才能满足实时流分析的需求。然而，并行和分布式计算系统的形状和大小多种多样，算法的设计应该与底层系统的特征相匹配。该项目开发了分析计算系统上的海量流数据的方法，范围从具有共享内存的多核的机器到通过广域网通信的地理分布的数据中心。这项研究的结果有望提高流分析的效率、延迟和吞吐量。由于所考虑的分析任务的基础性，本项目的结果将影响使用大规模机器学习和图形分析的学科，包括网络安全、社交网络分析和交通运输。生成的软件将作为工具包在流处理平台上发布，并部署在智能城市摄像头基础设施中。绩效指标的研究目标和教学目标之间的协同作用将导致在现有课程中编写新的教学材料，并开发新的数据分析课程。来自任职人数不足群体的个人将被纳入该项目。该项目将受益于学术界、产业界和国家实验室在流分析方面的合作，并加强它们之间的合作。该项目的第一个技术重点是设计用于数据流计算的共享内存并行算法，该算法可以实现复杂分析任务的高吞吐量和快速收敛。第二个重点是设计分布式流传输算法，通过在结果的新鲜度和通信量之间找到良好的折衷，能够容忍可变的通信延迟并适应广域网中的可用带宽。这些进展将在基本的图形分析和机器学习任务的背景下进行研究，例如子图计数、图形连通性和聚类、矩阵因式分解和深度网络。该项目将利用并行计算领域开发的大量理论和技术来设计处理流数据的方法，从而产生可在不同应用程序中重复使用的技术工具包。该项目还将在顺序流和针对某些问题的增量算法方面取得进展；例如，机器学习中使用迭代收敛方法的问题。基于设计的技术，该项目将设计和构建一个分层参数服务器，该服务器可以在从多核机器到数据中心再到广域数据源的范围内有效运行。

In today's big data era, there is an urgent need for methods that can quickly derive analytical insights from large volumes of data that are continuously generated. Such streaming data include video, audio, activity logs, and sensor data, and are generated on a massive scale all over the world. The need for real-time streaming analytics can only be fulfilled with the help of appropriately designed parallel and distributed algorithms. However, parallel and distributed computing systems come in a variety of shapes and sizes, and algorithms should be designed to match the characteristics of the underlying system. This project develops methods for analyzing massive streaming data on computing systems ranging from machines with multiple cores sharing memory to geo-distributed data centers communicating over wide-area networks. The results of this research are expected to improve the efficiency, latency, and throughput of streaming analytics. Due to the foundational nature of the analytical tasks considered, results of this project will impact disciplines that use large-scale machine learning and graph analytics, including cybersecurity, social network analysis, and transportation. Resulting software will be released as toolkits on stream processing platforms, and deployed in a smart-city camera infrastructure. Synergy between the research goals and the teaching goals of the PIs will lead to new instructional material in existing courses as well as development of new courses in data analytics. Individuals from underrepresented groups will be included as a part of the project. The project will benefit from and strengthen collaborations between academia, industry, and national labs on streaming analytics. The first technical thrust of the project is on designing shared memory parallel algorithms for computation on data streams, that can achieve a high throughput and fast convergence for complex analytics tasks. The second thrust is on designing distributed streaming algorithms that can tolerate variable communication delays and adapt to available bandwidth in a wide-area network, through identifying good tradeoffs between freshness of results and volume of communication. These advances will be studied in the context of fundamental graph analytics and machine learning tasks such as subgraph counting, graph connectivity and clustering, matrix factorization, and deep networks. The project will utilize the vast body of theory and techniques developed in the realm of parallel computing in the design of methods for processing streaming data, leading to a toolkit of techniques that can be reused across applications. The project will also lead to advances in sequential streaming and incremental algorithms for certain problems; for instance, problems in machine learning that use iterative convergent methods. Based on the techniques designed, the project will design and build a hierarchical parameter server that operates effectively across the spectrum from multicore machines to data centers to wide-area data sources.